Behavioral prediction and boundary settings, control and safety assurance of ml &amp; ai systems

ABSTRACT

Typical autonomous systems implement black-box models for tasks such as motion detection and triaging failure events, and as a result are unable to provide an explanation for its input features. An explainable framework may utilize one or more explainable white-box architectures. Explainable models allow for a new set of capabilities in industrial, commercial, and non-commercial applications, such as behavioral prediction and boundary settings, and therefore may provide additional safety mechanisms to be a part of the control loop of automated machinery, apparatus, and systems. An embodiment may provide a practical solution for the safe operation of automated machinery and systems based on the anticipation and prediction of consequences. The ability to guarantee a safe mode of operation in an autonomous system which may include machinery and robots which interact with human beings is a major unresolved problem which may be solved by an exemplary explainable framework.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority from U.S. patent application Ser. No.17/525,602, filed Nov. 12, 2021, entitled “BEHAVIORAL PREDICTION ANDBOUNDARY SETTINGS, CONTROL AND SAFETY ASSURANCE OF ML & AI SYSTEMS,”which claims priority from U.S. Provisional Patent Application No.63/113,445, filed on Nov. 13, 2020, the entire contents of which arehereby incorporated by reference.

FIELD

An exemplary embodiment relates to the field of machine learning andartificial intelligence systems.

BACKGROUND

Systems based on machine learning, such as autonomous systems andsemi-autonomous systems, may provide a beneficial decision-makingprocess. In a typical autonomous system, decisions that are usuallytaken by a human being are taken by the system itself. In asemi-autonomous system, decisions that are usually taken by a humanbeing are taken by the system, but the human beings can monitor thesystem and override decisions.

Referring to the embodiment of the prior art illustrated in FIG. 1 ,Zhou (2018) utilizes machine learning algorithms to construct models fortriaging failure events of autonomous vehicle systems. Zhou (2018)extracts features 110 from vehicle data logs 100, where the vehicle datalogs 100 are generated by an autonomous vehicle system. The extractedfeatures are used as input to machine learning classifier models 120, toclassify the type of failure 130 for the corresponding failure event,using the respective objective function 140. The features used in themachine learning classifier models may include the velocity,positioning, acceleration, and stopping power of the autonomous vehicle.The trained machine learning classifier models may assist in vehiclesystem failure events based on the features selected for the models.

Haynes et al. (2019) proposes an autonomous system that uses machinelearning models to detect static objects, where objects that areunlikely to move in the near future are classified. The machine learningclassification model (Haynes et al., 2019) may be trained using featuresprocessed from the sensor data and objects that are not detected asstatic are passed to additional machine learning models, such as aballistics motion model, to determine their trajectory prediction usingtraditional machine learning algorithms such as decision-tree basedmodels and neural network. The data retrieved from the sensors mayinclude the location of the nearby objects, such as traffic signals,additional vehicles, and pedestrians. The autonomous system proposed byHaynes et al. (2019) may determine a motion plan for the vehicle usingthe predicted values from the machine learning models. The motion planmay be used to control the motion of the autonomous vehicle.

Machine learning based systems may face challenges when it comes toautonomous and semi-autonomous systems. Autonomous and semi-autonomoussystems were the cause of fatal accidents such as the Tesla autopilotincident in May 2016 and the Uber incident in March 2018. Theseincidents highlight the challenges regarding the safety of artificialintelligence in autonomous and semi-autonomous systems. Methodologiesare required to assure the safety of machine learning systems in orderto take the necessary actions and avoid fatal incidents.

Referring now to FIG. 2 , FIG. 2 may illustrate the process flowregarding the behavior of autonomous and semi-autonomous systems. Theabstract process flow of an autonomous system may consist of designerrors 310, which may affect the intended behavior 300 of the autonomousor semi-autonomous system and hence alter the actual outcome 320 of theautonomous or semi-autonomous system. Errors in the design of a system310 may include the lack of critical real-time actions, based on thepredictions generated by a machine learning model, such as the lack oftimely activation of the emergency brake in a self-driving car system,in a situation that could lead to a fatal accident or similar impactfulaccident or event. In a semi-autonomous system, a human operatorreceives the actual behavior continuously 350 in order to understand thebehavior of the system 340. The documentation of the system 330 may alsoprovide an understanding of the behavior of the system 340. The humanoperator may observe the behavior of the system 350 and the actualdesign of the system 300, in order to make the necessary decisions inresponse 360 to the actual behavior of the system and in certain casesmay override the decisions made by the semi-autonomous system.

Watchkeeper is an Unmanned Air System (UAS) utilized by the UK army,consisting of a semi-autonomous system operated by human operators atthe Ground Control Station (GCS) which was the initial subject ofMcDermid et al. (2019). Watchkeeper has experienced five accidents todate which were caused by an incomplete understanding of the integrationof sub-systems of the semi-autonomous system.

The safety framework proposed by McDermid et al. (2019), as shown inFIG. 3 , illustrates the gaps of the safety and assurance in anautonomous system that utilize machine learning models for decisionmaking. This framework describes safety gaps that are created due to theinvolvement of machine learning in an autonomous system, wheretraditional safety principles would not apply to these gaps.

The framework proposed by McDermid et al. (2019) may include four maincomponents: the real world 400, the world as imagined 420, the safetyand assurance policies 440 and the world as observed 450. The real worldrefers to the environment of the operation of the system when it isdeployed. The world as imagined 420 may refer to the modelling 425 ofthe system based on the perception of the real-world environment by thedesigners of the system. Hence, the safety analysis 430 of the world asimagined 420 may be limited to the design precautions that wereimplemented by the designers. The world as observed 450 may refer to thedata produced in real-time by the system, such as images from sensorsand prediction output from machine learning algorithms. Safety andassurance 440 cases may be initially based on the world as imagined,however may be constantly updated by the world as observed to reflectthe safety measures on the real-world environment 405.

The framework (McDermid et al., 2019) may be deployed in operation inorder to continuously update each state to minimize the gap between theworld as imagined 420 and the world as observed 450. The gap may beminimized by reducing the possibility that the autonomous system wouldnot function as intended. FIG. 3 may illustrate an exemplary frameworkas known in the prior art. The framework includes four main components:the real world 400, the world as imagined 420, the safety and assurancepolicies 440 and the world as observed 450. The real world 400 refers tothe environment 405 that the autonomous system 410 is operating in. Theworld as imagined 420 refers to the modelling 425 of the system and thesafety analysis 430 based on these models.

The world as observed 450 refers to the data produced in real-time bythe system, such as images from sensors and prediction output frommachine learning algorithms, including the model used at runtime and theruntime data 455 and the ML analysis 460. The Safety and Assurance 440policies are initially based on the world as imagined 420; however, suchpolicies are constantly updated by the world as observed 450 to reflectthe safety measures on the real-world environment 405.

The framework proposed by McDermid et al. (2019) shows the gaps in thesafety and assurance measures that typical machine learning models facein an autonomous system. The gap between the real world 400 and world asimagined 420 illustrates the assumptions that the data analysts usuallymake during the construction of the machine learning models. Assumptionsmay include statistical assumptions, features selected for trainingdata, and distribution of the training data. The statistical assumptionsin the world as imagined 420 may not apply to the world as observed 450.The features selected to build a machine learning model may not beavailable in the world as observed 450 and hence an autonomous systemmay end up with observations 455 with too few features that areinsufficient to generalize a prediction output correctly for decisionmaking.

This framework highlights the gaps between the performance of the designand simulation of system components designed for an imagined environmentwith the performance on the observed environment. The gap between realworld and world as imagined highlights the assumptions the data analystsmake during the construction of the machine learning models. Theseassumptions may include statistical assumptions, features selected fortraining data, and distribution of the training data. Statisticalassumptions in the world as imagined may not apply to the world asobserved.

The gap between the real world 400 and the world as observed 450 mayinclude sensor limitations with respect to the environmental conditions,limitations of the machine learning algorithms such as false positivesand false negative observations and the limitations of human cognitiveability in semi-autonomous systems to respond to the output of thesystem.

The safety analysis 430 is based on hypothetical assumptions and thehuman understanding of the real-world environment 405. However, someautonomous systems that are based on machine learning algorithms may berich in data generation. Data allows the understanding of the influenceof the features regarding safe behavior and therefore may provideexplainability on the features. The influence of the features, orfeature importance, can allow for the construction of rules to limitcertain behaviors of the autonomous system to stay within certain safeboundaries.

In order to achieve safety in critical scenarios such as in autonomoussystems where certain tasks may be fatal to a human being, assurance maybe required at each step of the machine learning life cycle. Ashmore etal. (2019) illustrates four stages, or machine learning components, of amachine learning model lifecycle, where each stage should be analyzedfor safety and assurance, namely: data management, model learning, modelverification and model deployment.

The data management stage may include dataset collection, required forthe training and validation of the machine learning model, and datapre-processing and feature engineering to create an optimal model. Thisstage produces the training dataset and the validation dataset.Collected data needs to be relevant to ensure system safety. In anexemplary embodiment, in an autonomous vehicle application, Chinese roadsigns would not be relevant when operating in Germany (and vice-versa).Ashmore also recommend that data exhibit the following properties:balanced sampled data for each class; domain suitability for the taskbeing solved across the input space; data completeness; and dataaccuracy.

The model learning stage may include the selection of the machinelearning algorithm in order to train the model using hyperparametertuning techniques, validation dataset and cross-validation techniques toavoid overfitting the model on an unseen dataset, and other relevanttechniques. Model parameters may be tuned to reduce generalization errorwhile reflecting the objective function of the task. Complex models mayrequire a considerable amount of time to train and may use transferlearning to transfer domain knowledge from one ML model to another MLmodel. In this work, explainable models may be initialized bytransferring knowledge from a black-box model, or by transferringknowledge from an existing ruleset, which has been trained externally,or directly from a human supplied ruleset. Transfer learning may improvethe speed and quality of the resulting explainable model. The resultingperformance of the ML model (as determined by suitable performancemetrics) also ensures the safety of the model learning component.Robustness, behavior in face of previously unforeseen data andout-of-distribution (OOD) data, and re-usability are also main factorsthat affect safety.

The model verification stage ensures that the performance of the trainedmodel is consistent on unseen data, including testing on new data,running verification tests for the validation dataset, performancetesting, formal verification, and other suitable tests. Detected errorsthat violate expected test criteria pre-established by the safety tests,may raise a flag for further refinement of the model, which may returnthe process all the way back to the data management stage.

The model deployment stage may include integration of the trained andverified model in a deployed system. Runtime checks, such as input datachecks, environmental monitoring, and monitoring of the internalcomputation state of the ML model may be needed to ensure safety in casean abnormal event modifies a safety condition of the model in a criticalsituation.

A model of a computation that includes a set of states and transitionfunctions between states is known as a finite state machine (FSM). AnFSM may include the start state where the computation begins and maythen transition to other defined states according to the transitionfunction. An FSM may be represented by a 5-tuple vector as defined inequation 1.

<Q,Σ,δ,q ₀,μ>  (1)

Q represents a set of states, Σ is a set of finite symbols that the FSMaccepts in the model, δ represents the transition function, q₀ is thestart state and μ represents the final states of the model, where μ⊆Q.

An FSM with probabilities for each transition between states is called aMarkov chain. These probabilities are known as transition probabilities.A Markov chain is a discrete-time stochastic process that makes use ofthe Markov property with a set of states Q. The Markov property isdefined where each future state s_(ƒ), where s_(ƒ)∈Q, is conditionallyindependent of the prior state given the current state. Conditionalindependence may be defined given two states u₁ and u₂ that areconditionally independent of an event g. The states u₁ and u₂ areindependent events in their conditional probability given g, as shown inequation 2.

P(u ₁ ∩u ₂ |g)=P(u ₁ |g)P(u ₂ |g)  (2)

A variant of the Markov chain where the current state is not observableis a Hidden Markov Model (HMM). An HMM, defined as shown in equation 3,generates the probabilities B, where each probability value refers tothe probability of an observation o_(k) from a state q_(i). Q representsa set of states, where Q={q₁, q₂, . . . , q_(N)}. O represents thesequence of observations, where each observation o_(i) is drawn from avocabulary V, where V={v₁, v₂, . . . , v_(n)}. A represents thetransition probability matrix, where the probability refers to moving,for example, from the current state q_(i) to the next state q₁. Π refersto the initial probability distribution over the states. An HMMinterpret its states, where such states are not directly observable, byanalyzing the pattern of a sequence of observed symbols from suchstates.

Q,A,O,B,Π>  (3)

Petri Nets may provide a graphical notation to describe complex systemsand processes. Petri Nets may be constructed as directed graphs.Exemplary Petri Nets may include five elements as shown in Equation 4. Pmay represent a finite set of n places. L may represent a finite set oftransitions. EV may represent the flow relation between P and thetransitions L. W may represent the weight mapping for EV and m_(o) mayrepresent the initial representation of P.

N=<P,L,EV,W,m _(o)>  (4)

Blind spots are critical in autonomous and semi-autonomous system sincethe input to these models may contain noise that may result in anincorrect prediction and thus, the safety of these systems might not beassured. Adversarial generated observations generate wrong predictionswith high confidence (Goodfellow et al., 2014) and highlight blind spotsin machine learning algorithms.

Current autonomous and semi-autonomous systems that are fully orpartially based on black-box machine learning models to predictinformation, such as the motion of detected objects, or for theprevention of a component failure in the system, are unable to providean explanation or justification for the predictions. Subsequently, suchcurrent systems are unable to detect the most prominent features, andunable to detect feature bias in an interpretable, evidenced, andauditable manner.

SUMMARY

According to at least one exemplary embodiment, a method, system, andcomputer program product for a behavioral model for safety and assurancein artificial intelligence (AI) and machine learning based systems maybe shown and described.

Machine learning (ML) based autonomous systems that use black-box MLmodels may be difficult to adjust for safety and assurance in areal-time scenario, as these systems can only adjust sub-components andtrigger actions based solely on the prediction output of the ML models.The nature of black-box models does not allow an autonomous system orsemi-autonomous system to interpret the model and provide an explanationon how it arrived at the predicted result. This information is criticalin autonomous systems. An autonomous system or semi-autonomous systemthat uses white-box ML models may provide an explanation of the impactof each input feature, and the internal coefficients, on the predictionoutput.

An exemplary embodiment introduces a behavioral modelling framework BM,as shown in FIG. 4 , for the autonomous systems that are based on anexplainable white-box model 500 x and/or causal architecture 510.Behavioral modelling framework BM may set conditional boundaryconstraints BM_(c), that upon activation, fire events BM_(e) to activatetriggers BM_(t), where the conditional constraints BM_(c) are based onthe coefficients of internal parts of the explainable white-box modelBM_(x) 500 and/or causal architecture 510. Conditional boundaries BM_(c)may reduce the gap between the real world 400 and the world as observed450 by adapting to the current environment of the autonomous system andupdating the safety and assurance analysis 440 in real-time.

An exemplary embodiment may process data in real time via white-boxmodels. White-box models can explain the predicted result bybacktracking the result to the input feature space and can construct animportance value, known as feature attribution, for each input feature.In an exemplary embodiment, the white-box model may be customizable andhence enabling coefficients inside the white-box model to be modifieddirectly using either human knowledge injection or autonomous systemknowledge injection. The customizability of a white-box model may offerincreased safety and assurance in the AI based autonomous systems andsemi-autonomous systems.

BRIEF DESCRIPTION OF THE FIGURES

Advantages of embodiments of the present invention will be apparent fromthe following detailed description of the exemplary embodiments thereof,which description should be considered in conjunction with theaccompanying drawings in which like numerals indicate like elements, inwhich:

FIG. 1 is an exemplary embodiment of an illustration of the trainingconfiguration for failure type classifications as found in the priorart.

FIG. 2 is an exemplary illustration of the behavior of autonomous andsemi-autonomous systems.

FIG. 3 is an exemplary embodiment illustrating a safety assuranceframework as found in the prior art.

FIG. 4 is an exemplary embodiment of a behavioral model for safety andassurance in ML-based Systems.

FIG. 5 is an exemplary embodiment of a high-level XNN architecture.

FIG. 6 is an exemplary embodiment of an exemplary XNN architecture.

FIG. 7 is an exemplary embodiment of an INN architecture.

FIG. 8 is an exemplary embodiment of a decision boundary for proximityevents.

FIG. 9 is an exemplary embodiment of an explainable behavioural modelframework.

FIG. 10 is an exemplary embodiment of a queueing system for events in anexemplary behavioral model framework.

FIG. 11 is an exemplary embodiment of a fast XNN architecture.

FIG. 12 is an exemplary embodiment of a Behavioral Model Hierarchy (BMH)framework.

FIG. 13 is an exemplary embodiment of a distributed XNN trainingarchitecture.

FIG. 14 is an exemplary embodiment of a feed-forward distributedarchitecture for events, triggers, and actions.

FIG. 15 is an exemplary embodiment of a feed-forward distributedarchitecture for events, triggers, and actions.

FIG. 16 is an exemplary embodiment of a Structural Causal Modelintegrated in a Behavioral Model.

FIG. 17 is an exemplary embodiment of an XRL Agent FSM/Markov Process.

FIG. 18 is an exemplary embodiment of a conditional constraint on anexplainable architecture.

DETAILED DESCRIPTION

Aspects of the invention are disclosed in the following description andrelated drawings directed to specific embodiments of the invention.Alternate embodiments may be devised without departing from the spiritor the scope of the invention. Additionally, well-known elements ofexemplary embodiments of the invention will not be described in detailor will be omitted so as not to obscure the relevant details of theinvention. Further, to facilitate an understanding of the descriptiondiscussion of several terms used herein follows.

As used herein, the word “exemplary” means “serving as an example,instance or illustration.” The embodiments described herein are notlimiting, but rather are exemplary only. It should be understood thatthe described embodiments are not necessarily to be construed aspreferred or advantageous over other embodiments. Moreover, the terms“embodiments of the invention”, “embodiments” or “invention” do notrequire that all embodiments of the invention include the discussedfeature, advantage, or mode of operation.

Further, many of the embodiments described herein are described in termsof sequences of actions to be performed by, for example, elements of acomputing device. It should be recognized by those skilled in the artthat the various sequences of actions described herein can be performedby specific circuits (e.g., application specific integrated circuits(ASICs)) and/or by program instructions executed by at least oneprocessor. Additionally, the sequence of actions described herein can beembodied entirely within any form of computer-readable storage mediumsuch that execution of the sequence of actions enables the at least oneprocessor to perform the functionality described herein. Furthermore,the sequence of actions described herein can be embodied in acombination of hardware and software. Thus, the various aspects of thepresent invention may be embodied in a number of different forms, all ofwhich have been contemplated to be within the scope of the claimedsubject matter. In addition, for each of the embodiments describedherein, the corresponding form of any such embodiment may be describedherein as, for example, “a computer configured to” perform the describedaction.

The terms interpretable and explainable may have different meanings.Interpretability may be a characteristic that may need to be defined interms of an interpreter. The interpreter may be an agent that interpretsthe system output or artifacts using a combination of (i) its ownknowledge and beliefs; (ii) goal-action plans; (iii) context; and (iv)the world environment. An exemplary interpreter may be a knowledgeablehuman.

An alternative to a knowledgeable human interpreter may be a suitableautomated system, such as an expert system in a narrow domain, which maybe able to interpret outputs or artifacts for a limited range ofapplications. In an exemplary embodiment, a medical expert system, orsome logical equivalent such as an end-to-end machine learning system,may be able to output a valid interpretation of medical results in aspecific set of medical application domains.

It may be contemplated that non-human Interpreters may be created in thefuture that can partially or fully replace the role of a humanInterpreter, and/or expand the interpretation capabilities to a widerrange of application domains.

There may be two distinct types of interpretability: (i) modelinterpretability, which measures how interpretable any form of automatedor mechanistic model is, together with its sub-components, structure,and behavior; and (ii) output interpretability which measures howinterpretable the output from any form of automated or mechanistic modelis.

Interpretability thus might not be a simple binary characteristic butcan be evaluated on a sliding scale ranging from fully interpretable toun-interpretable. Model interpretability may be the interpretability ofthe underlying embodiment, implementation, and/or process producing theoutput, while output interpretability may be the interpretability of theoutput itself or whatever artifact is being examined.

A machine learning system or suitable alternative embodiment may includea number of model components. Model components may be modelinterpretable if their internal behavior and functioning can be fullyunderstood and correctly predicted, for a subset of possible inputs, bythe interpreter. In an embodiment, the behavior and functioning of amodel component can be implemented and represented in various ways, suchas a state-transition chart, a process flowchart or process description,a Behavioral Model, or some other suitable method. Model components maybe output interpretable if their output can be understood and correctlyinterpreted, for a subset of possible inputs, by the interpreter.

An exemplary machine learning system or suitable alternative embodimentmay be (i) globally interpretable if it is fully model interpretable(i.e., all of its components are model interpretable), or (ii) modularinterpretable if it is partially model interpretable (i.e., only some ofits components are model interpretable). Furthermore, a machine learningsystem or suitable alternative embodiment, may be locally interpretableif all its output is output interpretable.

A grey-box, which is a hybrid mix of a black-box with white-boxcharacteristics, may have characteristics of a white-box when it comesto the output, but that of a black-box when it comes to its internalbehavior or functioning.

A white-box may be a fully model interpretable and output interpretablesystem which can achieve both local and global explainability. Thus, afully white-box system may be completely explainable and fullyinterpretable in terms of both internal function and output.

A black-box may be output interpretable but not model interpretable, andmay achieve limited local explainability, making it the leastexplainable with little to no explainability capabilities and minimalunderstanding in terms of internal function. A deep learning neuralnetwork may be an output interpretable yet model un-interpretablesystem.

A grey-box may be a partially model interpretable and outputinterpretable system and may be partially explainable in terms ofinternal function and interpretable in terms of output. Thus, anexemplary grey-box may be between a white-box and a black-box on a scaleof most explainable and interpretable (white-box) to least explainableand interpretable (black-box). Grey-box systems may have a level ofmodular interpretability since some of their components may be modelinterpretable.

An explainable architecture x, where x∈{XAI, XNN, XTT, XRL, XSN, XMN,INN} or logically equivalent or similar architectures, may be integratedin behavioral model BM as part of the structure of an exemplary model. Abehavioral model BM may include conditions BM_(c), events BM_(e),triggers BM_(t) and actions BM_(a) based on attributions and informationthat is retrieved from internal states of the explainable architecturesor from the output of the computation from explainable architecturesBM_(x). The output may include any related meta information of theexplainable architectures.

Exemplary embodiments of explainable architectures that may beintegrated in the behavioral model BM include, but are not limited to,eXplainable artificial intelligence (XAI) models, Interpretable NeuralNets (INNs), eXplainable Neural Nets (XNN), eXplainable TransducerTransformer (XTT), eXplainable Spiking Nets (XSN) and eXplainable MemoryNets (XMN) models. A further exemplary embodiment may present methodsfor detecting bias both globally and locally by harnessing the white-boxnature of eXplainable Reinforcement Learning (XRL).

Although some examples may reference one or more of these specifically(for example, only XAI or XNN), it may be contemplated that any of theembodiments described herein may be applied to XAIs, XNNs, XTTs, XSNs,or XMNs interchangeably. Another exemplary embodiment may relate to biasdetection in Interpretable Neural Networks (INNs) and related grey-boxmodels, which may be a hybrid mix between a black-box and white-boxmodel. An exemplary embodiment may apply fully to the white-box part ofthe grey-box model and may apply to at least some portion of theblack-box part of the grey-box model. It may be contemplated that any ofthe embodiments described herein may also be applied to INNsinterchangeably.

Exemplary embodiments may also be implemented entirely in hardware usinga dedicated fixed hardware circuit such as digital electronic circuitry,analog circuitry, a digital-analog hybrid, integrated circuitry,application specific integrated circuits (ASICs), field programmablegate arrays (FPGAs), neuromorphic circuits, optical circuits,optical-electronic hybrid, and quantum computing hardware. Mixtures ofdedicated hardware and software and more general CPU based solutions maybe contemplated.

XNNs are a new type of Artificial Neural Networks (ANNs) that areinherently interpretable and explainable. The main concept behind an XNNis that the inner network structure is fully interpretable.Interpretability is built within the architecture itself, yet itfunctions like a standard neural network. This eliminates the need toapply additional techniques or processing for interpreting the result ofa neural network. XNNs compute both the answer and its explanation in asingle feed-forward step without any need for simulations, iterations,perturbation, etc. XNNs are also designed to be easily implementableboth in software but also in hardware efficiently, leading tosubstantial speed and space improvements.

The architecture behind an XNN may combine multiple local models intoone global model. Local models may analyze a small area within theentire search space. In an exemplary embodiment, when a transaction isanalyzed in a local manner, a linear model may sufficiently explain themodel. On the other hand, global models may provide an understanding ofthe model with a holistic view. XNNs may merge the two; multiplepartitions may represent the local zones and multiple linear models mayexplain each partition, which are combined to create a global model.Additionally, XNNs may be designed to cater for non-linear data byembedding transformations within the neural network itself, while stillretaining explainability. Each layer, neuron, and connection within anXNN has a precise and well known and understandable function, unlike astandard ANN which is a black-box. XNNs may give rise to new category ofneural networks that are understandable and interpretable.

Referring now to exemplary FIG. 5 , FIG. 5 may illustrate a schematicdiagram of an exemplary high-level XNN architecture. An input layer 600may be inputted, possibly simultaneously, into both a conditionalnetwork 610 and a prediction network 620. The conditional network 610may include a conditional layer 612, an aggregation layer 614, and aswitch output layer (which outputs the conditional values) 616. Theprediction network 620 may include a feature generation andtransformation 622, a fit layer 624, and a prediction output layer(value output) 626. The layers may be analyzed by the selection andranking layer 628 that may multiply the switch output by the valueoutput, producing a ranked or scored output 630. The explanations andanswers may be concurrently calculated by the XNN by the conditionalnetwork and the prediction network. The selection and ranking layer 628(roughly corresponding to component 740 in FIG. 6 ) may ensure that theanswers and explanations are correctly matched, ranked and scoredappropriately before being sent to the output 630 (roughly correspondingto component 760 in FIG. 6 ).

The processing of the conditional network 610 and the prediction network620 is contemplated to be in any order. Depending on the specificapplication of the XNN, it may be contemplated that some of thecomponents of the conditional network 610 like components 612, 614 and616 may be optional or replaced with a trivial implementation. Dependingon the specific application of the XNN, it may further be contemplatedthat some of the components of the prediction network 620 such ascomponents 622, 624 and 626 may be optional or replaced with a trivialimplementation.

It may be contemplated that in some circumstances, the selection andranking layer 628 and the output 630 may be combined together into oneintegrated component. For optimization purposes, the XNN may also beimplemented with both the conditional network 610 and the predictionnetwork 620 together with all their components merged into one network.This merged conditional and prediction network may also be merged with acombined selection and ranking layer 628 and the output 630. Thisoptimization may still result in a logically equivalent XNN, which maybe faster for feed forward processing but may suffer when it comes totraining via backward propagation and gradient descent techniques.

The XNN can thus be implemented in a way that there is the input layer600, and a combination of the conditional network 610 and the predictionnetwork 620, including the conditional layer 612, aggregation layer 614,switch output layer 616, feature generation and transformation layer622, fit layer 624, prediction layer 626, and ranking layer 628 leadingto the output 630. This combination may apply to all embodiments andimplementations of the XNN, including both software and hardwareimplementations. The transformative capabilities of XNNs in this regardare unique and unparalleled in other neural network implementationssince the white-box nature of XNNs allows flexibility and extrememerging to be performed without affecting the logical behavior of theXNN, although this can affect various attributes of a practicalimplementation, such as size/space usage, performance, resource usage,trainability, and overall throughput.

Referring now to FIG. 6 , FIG. 6 may illustrate an exemplary XNNarchitecture which combines the results from the switch output layer andthe value output layer. The example depicted in FIG. 6 is logicallyequivalent to the following exemplary ruleset:

${f( {x,y} )} = \{ \begin{matrix}{{{Sigmoid}( {\beta_{0,0} + {\beta_{1,0}x} + {\beta_{2,0}y} + {\beta_{3,0}x^{2}} + {\beta_{4,0}y^{2}} + {\beta_{5,0}{xy}}} )},} & {x \leq 10} \\{{{Sigmoid}( {\beta_{0,1} + {\beta_{1,1}x} + {\beta_{2,1}y} + {\beta_{3,1}x^{2}} + {\beta_{4,1}y^{2}} + {\beta_{5,1}{xy}}} )},} & {{x > 10} \land {x \leq 20}} \\{{{Sigmoid}( {\beta_{0,2} + {\beta_{1,2}x} + {\beta_{2,2}y} + {\beta_{3,2}x^{2}} + {\beta_{4,2}y^{2}} + {\beta_{5,2}{xy}}} )},} & {{x > 20} \land {y \leq 15}} \\{{{Sigmoid}( {\beta_{0,3} + {\beta_{1,3}x} + {\beta_{2,3}y} + {\beta_{3,3}x^{2}} + {\beta_{4,3}y^{2}} + {\beta_{5,3}{xy}}} )},} & {{x > 20} \land {y > 15}}\end{matrix} $

The ruleset may be found following the activation function 780. Theexemplary architecture in FIG. 6 may begin with an input 700. The inputmay then be used as inputs to the conditional network 710 and theprediction network 720. The prediction network 720 may contain a featuregeneration and transformation layer 722, a fit layer 724, and a valueoutput layer 726. The value output layer 726 may provide equations whichcorrespond to rules which weigh different features of the inputs.Further, the input 700 may be used as input to the conditional network710. Again, the conditional layer 712 and aggregation layer 714 mayproduce conjunctive rules or other logical equivalents or partitionswhich are represented in the switch output layer 716.

The outputs of the value output layer 626 and the switch output layer616 may be combined 740 in the output layer 630. Once the output layer630 has been formed, a sigmoid or other activation function 780 may beapplied to the result 760, depending on the application.

XNNs may present an intuitive way to construct interpretable models,while still utilizing the power of ANNs and related methods such as deeplearning. Once the model is trained through back-propagation or asimilar method, the resulting neural network can be used to servepredictions and the inner structure of the XNN can be used to constructthe rules.

XRL introduces an explainable reinforcement learning system. Anexemplary embodiment may be based on the Bellman equation. XRLintroduces explanations to the actions and the environment where the XRLsystem is deployed. An action may refer to the input provided to theenvironment, calculated by applying a policy to the current state. Thismay be discrete or continuous. The set of all possible actions is calledthe action space.

FIG. 17 shows the XRL agent FSM/Markov Process. An exemplary embodimentmay include an action selection and ranking 1800, that is, some action afor the current state s, which leads to state s′. The reward is denotedby r. The XRL agent may have a simulation of the environment used in theaction selection process. The model may have additional connectionpoints, depending on the structure of the model itself. The XRL actormodel may be implemented as an exemplary concurrent computation unit. AnXRL actor may include a message queue and an internal state, where themessage queue may contain messages received from other actors and theprivate state may be updated by the processing of a message. In anexemplary embodiment, internal states and message passing behavior canthen be represented using various means including process calculus suchas CSP, CCS, ACP, LOTOS, π-calculus, ambient calculus, PEPA, fusioncalculus and the join-calculus. Such XRL actor models can have theirbehavior modelled using a combination of parallel composition ofbehaviors, communication and transmission models, sequential compositionof behaviors, reduction and interpretation rules, and abstraction rulesthat hide complexity and inner workings of a component from othercomponents.

An actor or agent may observe an ongoing situation in real-time andreceive information with regards to the positioning and the orientationof the relevant objects. The agent observes the state of the world it isperceiving and abstracts it into a state representation. The staterepresentation is used to predict the next state of the agent. Anexplainable agent or explainable aware agent may also utilize acombination of explanation, interpretation, and justification in itsprediction.

An exemplary XRL modification introduces explanations x as part of themodel/environment model. The world model may return a partial or fullexplanation regarding the state s′ and the reward r, which defined asx_(e). Another modification may be in the action space, which introducesan associate explanation, a, x_(a) which denotes an action andexplanation of the action, respectively. A policy may refer to themapping from a past experience to an action. The policy Π, in XRLbecomes Π_(x), which is now an explainable mapping, such that:

Π_(x) ≈s→a,x _(a)

Π_(x) ≈s,x _(s) →a,x _(a)

In terms of behavioral FSM, each (state, action) pair can have aconstant connection point prior to making the action, after selectingthe state, during the action, and after the action is made. Forreinforcement learning and XRL, another connection point may be before,during, and after the selection of the action under a policy π. This maybe applicable when the action space and/or the state space is eitherdiscrete or continuous. Explanations in the XRL learning process maylead to better safety and control mechanisms since they may allow forbetter understanding of the inner working of the system which mayrequire adjustments, monitoring, and automatic/manual interventions.

XTTs, may provide an alternative embodiment which uses: (i.) one or moretransducers in a pipeline that outputs the answer together with anexplanation as part of the transduction process, and/or (ii.) a suitabletransformer architecture, that may optionally be a replacement for gatedrecurrent neural networks or similar types of machine learning models,where the attention mechanism is extended to cater to the creation ofthe explanation alongside the answer. The encoder part of thetransformer encodes information about which parts of the input data arerelevant to each other, together with information about what parts ofthe explanation data are relevant to each other and encodes this data ina latent space that includes both the answer and the explanation. Thedecoder part of the transformer decodes the encodings while using theattention mechanism to construct and then output both the answer and itsassociated explanation. It is contemplated that alternative embodimentsmay, for example, use separate latent spaces for the answer and theexplanation, and other logical modifications may be found that may beamenable for practical and efficient implementations, especially forlarge scale parallel processing. Hardware deployments may also becontemplated.

Referring now to the exemplary embodiment in FIG. 7 , FIG. 7 mayillustrate an exemplary INN architecture. INNs may provide anarchitecture which can automatically generate an explanation usingexisting deep learning techniques. INNs may utilize existing softwareinfrastructures and hardware used for neural networks and may alsoremain fully compatible with backpropagation training techniques.

An exemplary INN architecture may include a feature transformer whichconverts the input to some hidden features, and a number of relevanceestimators which transform the hidden features to feature weights. Thefeature weights are then combined with the transformed input in order toextract the attribution of each input transformed feature. The resultingfeature attributions may then be aggregated for the result. Featureattribution may be extracted at different levels. In the simplest form,attribution may be linked directly with the original inputs. In othercases, such as CNNs, feature attribution may also be computed forhigher-level features which are typically found in kernels and filters.Additionally, INNs may split the model into various partitions, thusenabling a higher-level of flexibility and interpretability, by enablinga mixture of local or segmented explainability. In some cases, INNs arealso capable of providing global explainability.

Referring to FIG. 7 , an exemplary INN architecture may start with someinput vector X 800. The input 800 may then connect to a featuregeneration and transformation network 802 and to k relevance estimators804. The transformed features may be abstract or high-level featureswhich may be computed using a deep neural network such as a CNN, anon-linear mathematical function such as polynomial expansion, or someother form of generated features, which may be discrete or continuous.An exemplary relevance estimator may calculate the coefficient, at leastin a local manner, of each transformed feature.

In mathematical terms, the transformation network may be denoted as afunction T(X). Similarly, θ_(j)(X) represents the relevance function ofthe jth partition. If X→T(X) returns a vector with z transformeddimensions, then X→θ_(j)(X) also returns a vector with z coefficients,or relevance weights. It may be assumed that |T(X)|=|θ_(j)(X)|=z.

INNs may be flexible enough such that complexity may be modeled throughvarious options and possible configurations. The functions X→T(X) andX→θ_(i)(X) may be a deep neural network which allow for modellingcomplex abstract features. It may be contemplated that the combinationof T(X) and θ_(i)(X) may represent various embodiments of explainablemodels which are possible to implement with the INN architecture.

An exemplary embodiment may include a conditional network 806, where theneural network handles the conditional aspect of the model. For example,an embodiment may evaluate rules in the form of IF-conditions in orderto activate one or more partition. The output of Ci(X) may be binary. Itmay be noted that the partitions may be static or dynamic and may bediscovered either through an external partitioning process or through aconnected neural network. It may also be noted that INNs may alsofunction with only one partition. For example, for all values of X,C_(i)(X) may always be one (1). This is equivalent to having zeropartitions. In this exemplary case, there is no need to apply apartitioning method to find suitable partitions.

In the feature attribution network 808, the neural network may computethe feature attribution of each transformed feature, which is activatedby the associated partition. The relevance attribution may multiply theresult of the computed coefficient with the transformed feature. Inmathematical terms, the feature attribution 808 may compute θ_(j)(X)T(X)for the jth partition. The output of layer 808 may serve as the basis ofthe explanation generation. The values from this layer may be used togenerate feature attribution graphs, heatmaps, textual explanations orother forms of explanations. It is further envisaged that other forms ofexplanations may be grouped and/or structured in the form of ExplanationStructure Models (ESMs).

In the aggregation layer 810, the neural network aggregates the resultsfor each partition. This may be the predictive result for the activatedpartition. In mathematical terms, the aggregation function may bedefined by A_(j)(θ_(j)(X)T(X)). In an exemplary embodiment, theaggregation function may be a simple summation of the featureattributions. This becomes equivalent to a linear function, at leastfunctioning in a local manner, such that the result R_(j)=θ_(j)(X)₁T(X)+. . . +θ_(j)(X)_(z)T(X).

Finally, the switch layer 820 may select the activated partition. Ifmore than one partition is activated, some ranking function 825 may needto be applied. The result may be generated through the result layer 830.

An exemplary embodiment may use causal modelling as part of anexplainable framework. Causal inference may measure the effect of causeson specific units. In an exemplary embodiment, a medical applicationwhere causes t and c are known and modelled using a causal DAG mayimplement causal inferences. The output variable y of a causal inferencemeasures the effect of the causes on a patient u, and can be illustratedas y_(t)(u) and y_(c)(u). The effect of the cause t on a patient urelative to cause c on a patient u can be measured usingY_(t)(u)−Y_(c)(u).

Coefficients may be extracted from the explainable model and used asinput to the causal inference model. The output of a causal model may beused to trigger an event or a terminal action 540 in the system.

Referring now to the exemplary embodiment in FIG. 13 , FIG. 13 mayillustrate an exemplary high-level architecture of a distributed XNNtraining system. A distributed explainable architecture DEA may beutilized in a behavioral model framework in order to increase theperformance of the defined models. DEA may contain multiple explainablearchitectures DEA_(x), such that x∈{XAI, XNN, XTT, XRL, XSN, XMN, INN}or logically equivalent or similar architectures, where thesearchitectures are processed in parallel. The number of explainablearchitectures 1410 in a distributed framework may be defined as DEA_(n).FIG. 13 illustrates a high-level architecture of a distributed trainingsystem, where DEA_(x) is the XNN architecture and DEA_(n) is n models.

DEA may split the dataset 1400 into multiple subsets 1410 of data inorder to train the explainable architectures DEA_(x). The models trainedin a DEA are aggregated 1420 during the training phase by calculatingthe average (or weighted average) from the parallel models. Theaggregate model may be formed based directly on the weights of themodel, rather than from the result of the individual models. Anexemplary DEA may be useful for large datasets where the training datacannot fit in the CPU/GPU memory of a single machine.

DEA may include hybrid models, such that the models in the architectureare a mix of x, where x∈{XAI, XNN, XTT, XRL, XSN, XMN, INN} or logicallyequivalent or similar architectures. An exemplary embodiment mayimplement multiple different models. In an exemplary embodiment, onedata part may implement an XNN while another data part of the samesystem may implement an XAI. The models may then be combined to createan aggregate model. The aggregate model may be equal to x, where x∈{XAI,XNN, XTT, XRL, XSN, XMN, INN} or logically equivalent or similararchitectures or may be a hybrid model implementing multiple differentmodels.

A DEA may incorporate multiple independent models where one model, oncetrained, can work independently without the need to rely on the fulldistributed architecture, which is optimized primarily for trainingpurposes. The models in a DEA may be a variant of the explainablearchitectures x. Variants may include convolutional XNNs (CNN-XNNs),predictive XNNs (PR-XNNs), text XTTs (T-XTTs), and logically equivalentor similar architectures.

In an autonomous system or semi-autonomous system, a behavioral model BMmay incorporate feedback actions BM_(aƒ) where BM_(af)∈{a_(ƒ,1), . . . ,a_(ƒ,n)} as an input to the underlying explainable architecture x, wherex∈{XAI, XNN, XTT, XRL, XSN, XMN, INN} or logically equivalent or similararchitectures. BM_(af) may represent feedback processes ofsub-components within the behavioral model system of the system or anupdate process that is received from the server of the behavioral systemBM.

An exemplary embodiment of a feedback process task may refer to anoutput from a trigger t in BM_(t) being used to update specific internalparts of the explainable architecture x in BM_(x). This may be possiblesince explainable architectures in BM_(x) are white-box models and thecoefficients and internal parts of the white-box models areinterpretable. This operation may not be possible when using black-boxmodels in the behavioral model BM.

Human knowledge injection (HKI) or system knowledge injection is anothertype of input in a behavioral model BM for autonomous systems orsemi-autonomous systems. The coefficients θ of an explainablearchitecture x within a behavioral model BM may be modified to enforcespecific rules. Rule enforcement may also be activated by a conditionalconstraint located in BM_(c), where BM_(c)∈{c₁, . . . , c_(n)}. Theactivation of a conditional constraint may fire an event e, where evente may activate a trigger t 530 where such rule enforcement is passedusing a feedback action 550 to the explainable model 500 or causalarchitecture 510.

In an exemplary embodiment, each environment in the framework proposedby McDermid et al. (2019) may be seen as a behavioral model BM asdescribed in FIG. 3 . The Real World 400 represents the Desired BehaviorModel (DBM). DBM represents the behavior the system is trying to achievewhile utilizing explainable architecture and causal modelling. Note thatthe DBM may not always represent the Real World 400. If the BM is beingapplied to the Real World as its environment, the DBM typically equatesto the Real World. However, in other scenarios, where differentenvironments may be utilized (controlled environments, simulatedenvironments, virtual environments, metaverse, etc.), the DBM may notequate to the Real World. DBM may be formally verified according to thegiven constraints in order to validate the system before being deployedin production. The observed behavioral model OBM 450 refers to thebehavioral model that is deployed in the system and may be observed bygathering information via runtime monitoring of this model. The expectedbehavioral model EBM refers to the behavioral model that is constructedbased on the world as imagined 420. The formal verifications,simulations and synthesis are based on the behavior that is imagined bythe designer of such system, based on the imagined scenarios of the realworld 400. The Safety and Assurance 440 constraints and guarantees maybe represented by a behavioral model framework and/or behavioral modelhierarchy that assures the safety, which may include conditionalconstraints and/or model boundaries, in the observed behavioral modelOBM and/or the expected behavioral model EBM.

In an exemplary embodiment, an autonomous vehicle behavioral modelsystem may be treating incoming objects from all directions equally, andthe system may be aware that a specific location needs specialattention. Hence, an event e may be fired as an input, either by a humanduring manual review of the system, or by the autonomous system itself.The event e may trigger a feedback action a. The action a may update therespective coefficients or create a rule or partition in the internalstate of the underlying model in order to minimize the gap between thedesired behavioral model DBM 400 and the expected behavioral model EBM420, by tuning the observed behavioral model OBM 450.

A BM may include an explainable architecture x. Input constraints may beincluded during the formal verification of the explainable architectureof such behavioral model BM_(x). Input constraints may be based on theinternal coefficients of the white-box model, or the featureattributions constructed for the input dimensions of observation o.Feature attributions may identify the importance of a given feature withrespect to the result. Hence, explainable architectures extend theverification process of behavioral modeling by allowing additionalconstraints to be designed on feature attributions in order to formallyverify the white-box model for potential bias detection in the system.

The output of an exemplary BM within an autonomous or semi-autonomoussystem may include a model interpretation to be used for an explainableupdate to the user of the system. The model interpretation may also beused to update the underlying explainable architecture BM_(x) or toupdate a sub-component within the autonomous or semi-autonomous system.

There may be three types of model interpretation: basic interpretation,explanatory interpretation, and meta-explanatory interpretation. A basicinterpretation may refer to a prediction output o that can be understoodby the sub-component. An explanatory interpretation may be representedby a 2-tuple vector <o, w> and may refer to a prediction output ocombined with a model explanation w for the predicted value, that can beunderstood by the sub-component. A model explanation may includecoefficients θ of the explainable architecture x that may be utilized toexplain the feature importance of the input features for a givenobservation. A meta-explanatory interpretation may be represented by a3-tuple vector <o, w, j> and may contain the prediction output o, themodel explanation w and justification of the model explanation j. Themodel justification j may provide additional information about theassumptions, processes and decisions taken by the explanation systemthat were taken into consideration to produce the model explanation.

In an exemplary embodiment, an event e is fired based on a certainconditional constraint c. The event e may activate a trigger t. Thetrigger t may send a model interpretation of the underlying explainablearchitecture x in the form of an informative update, such as displayinga warning on the user interface, to the user using the autonomous orsemi-autonomous system. An example of a warning can be that a vehicle isapproaching the autonomous vehicle at an unusual acceleration.

In an exemplary embodiment, the behavioral model BM of an autonomous orsemi-autonomous system may include singular or multiple directactionable output BM_(a), where a∈{a₁, . . . , a_(i)}. An actionableoutput BM_(a) may stop the vehicle or switch lanes to avoid a fatalaccident. The autonomous system would take a direct action, using thevehicle controller, to modify the autonomous system properties in orderto avoid a scenario.

The behavioral model of an autonomous or semi-autonomous system mayinclude an event stream pipeline P_(e), where e∈{e₁, . . . , e_(n)}.Pipeline P_(e) may include multiple events that may be fired from thesame sub-component. The final output of pipeline P_(e) may be a terminalaction a_(t) or a feedback action a_(ƒ) which may be used to update thecoefficients of the internal structure of an explainable architecture x.

A BM of an autonomous system or semi-autonomous system may contain a setof actions, where BM_(a)∈{a₁, . . . , a_(n)}, that may perform a changeto the status of a sub-component within the BM or raise an event ewithin the behavioral model BM. Actions BM_(a) may be triggered by atrigger t. A BM may contain a set number of triggers BM_(t)∈{t₁, . . . ,t_(n)}. A trigger t may be activated when a condition c set for anexemplary trigger, t_(c), is set to true. A trigger t may have multiplesets of conditions to be activated, such that t_(c)∈{c₁, . . . , c_(n)}.The trigger t may have a recency and frequency attribute that may eitherincrement or decrement the triggering rate according to the activationhistory of the trigger t. In a practical exemplary embodiment within theaviation industry, a trigger t may be defined as: If the conditionalconstraint c “altitude is below threshold β” is true, a feedback actiona_(ƒ) should trigger.

The activation of a conditional constraint c may fire an event e toactivate a trigger t. A particular conditional constraint c set on theExplainable System 570 may have multiple events associated with it, suchthat c_(e)∈{e₁, . . . , e_(n)}. In an exemplary embodiment, aconditional constraint c is set on a coefficient in the explainablearchitecture 1900, as shown in FIG. 18 , and upon activation, fires anevent e 1910 to activate a trigger t 1920 in order to activate an actiona 1930. It is contemplated that triggers, events and conditions mayimplemented using a pipeline or stream.

Referring now to the exemplary embodiment in FIG. 9 , FIG. 9 mayillustrate a schematic flowchart of an instance of an exemplaryexplainable behavioral model framework. A condition c may be aconstraint statement that may be set on the internal coefficients of theexplainable architecture x 500, where x∈{XAI, XNN, XTT, XRL, XSN, XMN,INN} or logically equivalent or similar architectures, the internalcoefficients of the causal model 510, or on any variable within thebehavior model BM. The activation of such conditional constraint c, maycause an event e 520 in FIG. 4 and event e 1021 or 1040 in FIG. 9 to befired to a particular trigger t 530 in FIGS. 4 and 1080 or 1031 in FIG.9 , in order to trigger 1055 a particular action a 540 or 550 in FIGS. 4and 1041, 1051 , or 1060 in FIG. 9 . An event e may trigger additionalevents within a BM, trigger a terminal action a_(t) 540 in FIGS. 4 and1041 or 1060 in FIG. 9 , or trigger a feedback action a_(ƒ) 550 in FIGS.4 and 1051 in FIG. 9A feedback action a_(ƒ), 550 in FIGS. 4 and 1051 inFIG. 9 , may trigger sub-components 1061 within a BM to perform aparticular task, execute an event e in an acyclical manner 1020 orexecute an event e in a cyclical manner 550, 560 as shown in FIG. 4 . Afeedback action a_(ƒ), 550 in FIGS. 4 and 1051 in FIG. 9 , may be usedas behavioral model knowledge injection to update internal parts of anexplainable architecture or causal architecture 1070 or 1030. Feedbackactions may also be handled by causal logics that can handle cyclicalcausal models, for example, using Input/Output SCMs (ioSCMs), temporalunrolling or other suitable methods. In an exemplary embodiment, aconditional constraint c may be set on a coefficient in the explainablearchitecture 1900, as shown in FIG. 18 , and upon activation, fires anevent e 1910 to activate a trigger t 1920 in order to activate an actiona 1930.

The triggers of a behavioral model BM_(t) may link its neuro-symbolicconditions with its previous historic rate of activations in order toconstrain the rate of trigger activation. In an exemplary embodiment, atrigger t_(i) may be based on the fusion of two conditions c_(n−1) andc_(n−2). An additional trigger t_(i−1) may be based on the conditionalconstraint c_(n−2). Hence, when an event is fired to activate thetrigger t_(i), trigger t_(i−1) is also activated, as the conditionalconstraint c_(n−2) was activated in order to activate the trigger t_(i).The fusion of conditions may be based on multiple models, within abehavioral model, such as causal model 510 and an explainablearchitecture x 500, where x∈{XAI, XNN, XTT, XRL, XSN, XMN, INN} orlogically equivalent or similar architectures, to trigger an internalaction 1051 or trigger a terminal action 1060.

A condition c may be a constraint statement that may be set on theExplainable System 570, such as on the internal coefficients of theexplainable architecture x 500, the internal coefficients of the causalmodel 510, or on any variable within the behavior model BM of suchautonomous system or semi-autonomous system. The activation of theconditional constraint c, may cause an event e 520 1040 to be fired to aparticular trigger t 530 1080 in order to trigger an action a 540 5501051 1060. A conditional constraint c may be constructed of a binaryconstraint, a signal constraint or be associated with an activationfunction in the underlying explainable architecture x. A condition c maybe based on other conditions in a hierarchical form. A condition may beof the form of conjunctive normal form (CNF), or disjunctive normal form(DNF), or a suitable first order logic in order to be compatible withformal verification problem solvers such as Satisfiability moduletheories (SMT) and conflict-driven clause learning (CDCL) Satisfiability(SAT) solvers.

In an exemplary behavioral model BM, a neuro-symbolic conditionalconstraint c, that is based on the Explainable System 570, such as onthe explainable architecture x 500, where x∈{XAI, XNN, XTT, XRL, XSN,XMN, INN} or logically equivalent or similar architectures, may fire anevent e, where event e may also be fired by a different conditionalconstraint on the causal inference architecture 510. This exemplaryembodiment eliminates redundant identical events from the behavioralmodel BM_(e). A trigger t may require multiple events to be received inorder for the trigger to be activated. This exemplary approach enablesmodularity of events BM_(e), conditional constraints BM_(c) and triggersBM_(t) in a behavioral model. It is further contemplated that similartechniques may be utilized to eliminate redundant components from the BMincluding a combination of events, triggers, conditions, and actions.

A neuro-symbolic constraint may be implemented in a variety of suitableexemplary implementations including, but not limited to, in the form ofsymbolic rules or system of symbolic expressions, polynomialexpressions, conditional and non-conditional probability distributions,joint probability distributions, state-space and phase-space transforms,integer/real/complex/quaternion/octonion transforms, Fourier transforms,Walsh functions, Haar and non-Haar wavelets, generalized L2 functions,fractal-based transforms, Hadamard transforms, Type 1 and Type 2 fuzzylogic, topological transforms of Kolmogorov/Frechet/Hausdorff/Tychonoffspaces, and difference analysis. Neuro-symbolic constraints may also beimplemented in form of a data structure that references the differentfeatures and variables accessible to the explainable model and anyassociated taxonomies, ontologies, and causal models. Neuro-symbolicconstraints may also be implemented in the form of knowledge graphnetworks.

The triggers of such behavioral model BM_(t) may link its neuro-symbolicconditions with its previous historic rate of activations in order toconstrain the rate of trigger activation. In an exemplary embodiment, atrigger t_(i) may be based on the fusion of two conditions c_(n−1) andc_(n−2). An additional trigger t_(i−1) may be based on the conditionalconstraint c_(n−2). Hence when an event is fired to activate triggert_(i), trigger t_(i−1) is also activated, as condition c_(n−2) wasactivated in order to activate trigger t_(i). Such fusion of conditionsmay be based on multiple models, within a behavioral model, such as acausal model and/or an explainable architecture x, where x∈{XAI, XNN,XTT, XRL, XSN, XMN, INN} or logically equivalent or similararchitectures, to trigger an internal action or trigger a terminalaction.

An event e 520 may trigger additional events or event sequences (alsoknown as an event cascade or event pipeline) within a behavioral modelBM, trigger a terminal action a_(t) 540 1060 or trigger a feedbackaction a_(ƒ) 550. A feedback action a_(ƒ) 550 may trigger sub-componentswithin a behavioral model BM to perform a particular task, execute anevent e in an acyclical manner or execute an event e in a cyclicalmanner in order to activate a particular trigger. A feedback actiona_(ƒ) 550 may be used as behavioral model knowledge injection to updateinternal parts of an explainable architecture 1030 or causalarchitecture 1070.

An event e may be relayed and processed by a message broker, as shown inFIG. 10 . A message broker may be an architectural pattern used toexchange messages effectively between components of a behavioral systemusing asynchronous communication between components. Events received1110 by the message broker are normally queued in a queueing system 1130that may be located in the message broker, and which utilizes a queuedata structure, such as a buffer, for the received messages. Events emay then be processed 1140 to the respective sub-components in thebehavioral model BM.

An event e that is attached to at least one conditional constraint c maybe known as bounded event e_(bounded). A concern event e_(concern) maybe a bounded event e_(bounded) that may refer to a safety concern in anautonomous system or semi-autonomous system. A concern event e_(concern)may raise awareness to the user of the system if a particular conditioncontinues to be true. A concern event e_(concern) might be bounded to aconditional proximity constraint c, as shown in FIG. 8 , and hence maybe compared to a confidence interval or boundary range around theboundary 930. The boundary range may be learnt using machine learningmodels and can be extended to any number of dimensions.

It is further contemplated that the boundary range may have anassociated set of tolerances and/or confidence intervals that allows forflexibility in the boundary range definition. It is further contemplatedthat the boundary range may have an associated boundary transformationfunction that dynamically transforms a combination of the boundaryrange, tolerances, confidence intervals or other suitable boundaryand/or boundary characteristic on an Explainable System 570 using asuitable transformation function such as a feedback loop control method,Nyquist control, Bode plot, fuzzy logic transform (Type 1, Type 2,Sugeno, Mamdani, etc.), transforms learnt via gradient descent methods,transforms specified via rule systems, first order logic, rotations,dimensional scaling, dimensionless scaling, Fourier transforms, Walshfunctions, state-space transforms, phase-space transforms, Haar andnon-Haar wavelets, generalized L2 functions, fractal-based transforms,Hadamard transforms, knowledge graph networks, categorical encoding,topological transforms of Kolmogorov/Frechet/Hausdorff/Tychonoff spaces,difference analysis, causal operators and other suitabletransformations.

A concern event e_(concern) that is bounded to a conditional proximityconstraint c may be utilized to raise several warnings or raise a redflag to the autonomous system or semi-autonomous system to execute aterminal action a in order to perform an action, such as to avoid afatal accident, for example. A conditional proximity constraint c, seton the internal components of an Explainable System 570, may be a binaryproximity, which refers to if the boundary range has been violated ornot. A violation of the conditional proximity constraint c may trigger afeedback action or a terminal action. A conditional proximity constraintc, set on the internal components of an Explainable System 570, may beprobabilistic or may include several values which represent differentstates in the behavioral system. The probabilistic or multi-valuedproximity constraint c may be attached to multiple triggers c_(t), wheret∈{t₁, . . . , t_(n)}.

The boundary of a conditional proximity constraint c set on the internalcomponents of an Explainable System 570, may be learnt using a machinelearning model. A hyperplane may be constructed and maximized using amachine learning model. The outcome values of the conditional proximityconstraint c may support a hyperplane 900 as illustrated in FIG. 8 . Ifa boundary is about to be transgressed 910, a concern event e_(concern)may be raised in order to trigger the appropriate action. The distance940 may be measured in the conditional space in order to fire an eventwith accurate representation of the current distance of the proximityconditional constraint to activate the respective action. The distanceremaining 950 for the conditional proximity constraint to betransgressed may be measured in order to fire an event with an accuraterepresentation of the current distance of the proximity conditionalconstraint to activate the respective action. A conditional proximityconstraint that has just been transgressed 920 may raise a concern evente_(concern) in order to trigger the appropriate action.

It is further contemplated that an action a arising from a conditionalproximity constraint c may dynamically transform a combination of theboundary range, tolerances, confidence intervals or other suitableboundary range characteristic on an Explainable System 570 using asuitable transformation function such as a feedback loop control method,Nyquist control, Bode plot, fuzzy logic transform (Type 1, Type 2,Sugeno, Mamdani, etc.), transforms learnt via gradient descent methods,transforms specified via rule systems, first order logic, rotations,dimensional scaling, dimensionless scaling, Fourier transforms, Walshfunctions, state-space transforms, phase-space transforms, Haar andnon-Haar wavelets, generalized L2 functions, fractal-based transforms,Hadamard transforms, knowledge graph networks, categorical encoding,difference analysis, causal operators and other suitabletransformations.

In an exemplary embodiment, an autonomous vehicle behavioral modelsystem may be treating incoming objects from all directions equally, andthis system may be aware that a specific location needs specialattention. Hence, an event e may be fired as an input, either by a humanduring manual review of the system, or by the autonomous system itself.The event e may trigger a feedback action a and such action a updatesthe respective coefficients or create a rule or partition in theinternal state of the underlying model, in order to minimize the gapbetween the desired behavioral model DBM 400 and the expected behavioralmodel EBM 420, by tuning the observed behavioral model OBM 450.

BMs may utilize causal logic including a combination of Pearl'sstructural causal models and associated derivations and variations,dynamic causal models and associated Bayesian model comparison methodsand variations, granger causal models, relativistic causal modelsarising from special and general relativity, and other suitableimplementations that allow machine learning to representcause-and-effect. In an exemplary embodiment, a BM may use causal logicto add cause-and-effect constraints on possible scenariointerpretations, increase overall safety and ensure a correctbehavioural response from an autonomous system that handles context in amore similar way to how humans handle context. It is furthercontemplated that the what-if, what-if-not and generic counterfactualprocessing (Rung 2 and Rung 3 of Pearl's Ladder of Causation) may beused to enhance the BM and BMH capabilities further.

Causation may be defined as a structural causal Model SCM in order todescribe the features of the datasets, being utilized by the model, andthe interactions between these features. A Structural causal Model SCMmay include three components: U, V and ƒ. U may refer to variables thatare external to the causal model and are not a descendant of any othervariables. U may refer to exogenous variables. V may refer to variablesthat are a descendant of at least one exogenous variable. V may refer toendogenous variables. The component ƒ may refer to the functions thatare utilized to derive V variables from the U variables.

A Structural causal Model SCM may be associated with a directed acyclicgraphical model. A graphical model G may contain N nodes and E edges.The graphical model G_(N) contains a node for each exogenous variable inSCM_(U), where U∈{U₁, . . . , U_(n)}, and a node for each endogenousvariable in SCM_(V), where V∈{V₁, . . . , V_(n)}. The edges G_(E) of anexemplary graphical model may refer to the functions used to derive theendogenous variables SCM_(V). The graphical causal model G may haveconditional constraints G_(c), where C∈{c₁, . . . , c_(n)}, set on thevalues of G_(N), such that if the values exceed certain threshold t, anevent e is fired to activate a trigger t. The trigger t may execute aterminal action or a feedback action to update internal coefficients ofa causal model, update internal coefficients of an explainablearchitecture x, or update a sub-component within the behavioral modelBM.

In an exemplary embodiment, as illustrated in FIG. 16 ,SCM_(U)={experience_(school),experience_(work)} 1740 1750,SCM_(V)={salary} 1760 and SCM_(ƒ)={ƒ_(salary)}, whereƒ_(salary):=(2*experience_(school))+(3*experience_(work)). As shown inFIG. 16 , the variables experience_(school) 1740 and, experience_(work)1750 are direct causes of the salary variable. A conditional constraintmay be based on the values of the experience_(school), experience_(work)or salary variables. A conditional constraint may be based onƒ_(salary), specifically on particular variables within the equation. Anexemplary behavioral model BM allows for the fusion of conditionalconstraints in order for the activation of a trigger t; hence multipleconditional constraints may be based on the graphical causal model 510and on the explainable architecture 500. An event 1700 may be fired onthe activation of conditional constraints for a particular trigger 1710within the behavioral model to execute a terminal action or a feedbackaction 1720 to update internal coefficients of a causal model 1730,update internal coefficients of an explainable architecture x, or updatea sub-component within the behavioral model BM.

In an exemplary embodiment, a BM will use a suitable computational andknowledge representation structure as the basis of its constraint andpredictive logic implementation. Such a suitable structure may be aResource Description Framework (RDF) tree, RDF graph, or other suitableform of graph structure. It is further contemplated that a hypergraphstructure or a simplicial complex may be used in a practical BMimplementation.

A BM may set conditions based on the global feature attributions of theinput dimensions of an explainable architecture x, in an ExplainableSystem 570. It is further contemplated that a BM may set conditionsbased on the local model feature attributions and/or the hierarchicalpartition structure of an explainable architecture x. In a generalizedformat, let m represent the number of input dimensions (example x, y)and some transform function Transform(X) takes a matrix of m dimensionsand returns a matrix with k transformed dimensions (for example, x, y,x², y², xy). Let C represent a matrix of coefficients where j representsthe total number of rules in the rule-based model.

$C = \begin{bmatrix}C_{0,0} & \ldots & C_{0,{k - 1}} \\ \vdots & \ddots & \vdots \\C_{{j - 1},0} & \ldots & C_{{j - 1},{k - 1}}\end{bmatrix}$

The matrix of coefficients may then be aggregated such that the vector Irepresents the importance of each feature from all j rules such thatI={θ₀, . . . , θ_(i), . . . , θ_(k−1)} where θ_(i)=Σ_(p=0)^(j−1)C_(p,i). Finally, let I_(s)={F₀, . . . , F_(s), . . . , F_(k−1)}represent a sorted vector with all elements of I where s represents thesort index, such that F_(s−1)≥F_(s)≥F_(s+1). A mapper vector M may alsobe used to link the sorted coefficient index s with the transformedfeature index k. A BM may create conditions BM_(c) based on fusionbetween matrix coefficients and the input dimensions of the currentobservation. It is further contemplated that other suitableimplementations of I, θ, F and M and/or any other part of the BM may bealternatively implemented to allow for logically suitable extensionssuch as Type 1 and Type 2 fuzzy logic systems and other suitable logicsystems that allow for behavioral modelling and/or specifications.Constraints and expressions underlying conditions, events, triggers andactions may be implemented in a variety of suitable exemplaryimplementations including, but not limited to, in the form of symbolicrules or system of symbolic expressions, polynomial expressions,conditional and non-conditional probability distributions, jointprobability distributions, state-space and phase-space transforms,integer/real/complex/quaternion/octonion transforms, Fourier transforms,Walsh functions, Haar and non-Haar wavelets, generalized L2 functions,fractal-based transforms, Hadamard transforms, Type 1 and Type 2 fuzzylogic, topological transforms of Kolmogorov/Frechet/Hausdorff/Tychonoffspaces, and difference analysis. Constraints and expressions may also beimplemented in form of a data structure that references the differentfeatures and variables accessible to the explainable model and anyassociated taxonomies, ontologies, and causal models. Constraints andexpressions may also be implemented in the form of knowledge graphnetworks. Other constraints and expressions may also be implemented onthe basis of useful heuristics, such as control charts, Nelson rules,Bode plots, Nyquist plots and related methods that determine whethersome measured behavioral variable is out of control—i.e., giving ameasure of unpredictability versus consistency, which may be highlyuseful in a practical implementation of a BM.

A BM may set conditions based on the local feature attributions of theinput dimensions of an explainable architecture x, in an ExplainableSystem 570. An exemplary input sample may have two inputs, in this casex and y. A feature importance vector I may represent the featureimportance in a local manner such that I={β₀, β₁, . . . , β_(n)}, wheren=|F| is the total number of transformed features corresponding to thegenerated features F. In an exemplary embodiment, F may include{x,y,xy,x²,y²}. Given a specific input vector {x, y}, it may becontemplated that one or more rules may trigger through the functionƒ(x,y). In this exemplary embodiment, let x≤10. This may trigger thefunction Sigmoid(β₀+β₁x+β₂y+β₃xy), which results in the localizedfeature importance vector I_(L)={β₁x,β₂y,β₃xy}. In an exemplaryembodiment, a condition BM_(c) may be set on the feature importancevector in order to trigger a bias warning to the interpreter of thebehavioral model.

The underlying explainable architecture x may contain a separatebehavioral model x_(BM) that is utilized during training, specificallyduring the backpropagation phase. The behavioral model x_(BM) may beseparate from the behavior model BM that is used on the deployedexplainable architecture. Hence, x_(BM) designed for backpropagationmode may include separate conditions, events, triggers, and actions.

A BM may contain multiple explainable architectures BM_(x), such thatBM_(x)∈{x₁, . . . , x_(n)}. The architectures BM_(x) may shareconditions c, triggers t, actions a and events e. In this exemplarycase, there might exist some conditions, triggers, actions, and eventsin x₁ and x₂ that are the same, referred to as “identical” in theformula, as shown in Equation 5, Equation 6, Equation 7 and Equation 8.

(x1,x2)∈BM _(x) ,∃k∈x1_(c) ,∃u∈x2_(c):identical(k,u)  (5)

(x1,x2)∈BM _(x) ,∃k∈x1_(t) ,∃u∈x2_(t):identical(k,u)  (6)

(x1,x2)∈BM _(x) ,∃k∈x1_(a) ,∃u∈x2_(a):identical(k,u)  (7)

(x1,x2)∈BM _(x) ,∃k∈x1_(e) ,∃u∈x2_(e):identical(k,u)  (8)

A named reference label may be assigned to particular components withinan explainable model and/or a behavioural model. Named reference labelsmay be descriptive in nature and may also contain additional meta-dataand links to external taxonomies, ontologies, and models. A namedreference label may consist of symbolic expressions and/or formulas ofthe form of conjunctive normal form (CNF), or disjunctive normal form(DNF), or a suitable first order logic, to provide an explanation of theset or sequence of decisions that resulted in the execution of thecurrent component, in the behavioral model BM, which we refer to as thepath trace. An “activation path” may be defined as a data flow pathfollowed by an AI model, starting from a particular input, and ending ina particular output prediction. The path trace is set or sequence ofdecisions, nodes, transitions, or applicable description structures forthe particular AI model describing the activation path. Named referencesmay be used in safety related constraints to ensure easier and morereliable identification by humans, especially in potentially criticaland stressful situations where human attention span may be severelytaxed or limited.

In an exemplary embodiment, named reference labels may contain meta-dataabout multimedia files associated with that named reference label, unitsand dimensions associated with the explainable model component, and soon. The named reference label itself is a direct form of symbolicreference, that can either be the named reference label text itself oran associated meta-data.

In an exemplary embodiment, the named reference labels themselves mayalso be used by a suitable model discovery system or model optimizationsystem, such as an exemplary AutoXAI system (that in this case, maydiscover and optimize BMs), to generate human-friendly explanations ofBM related or other dynamical processes that may be otherwise verydifficult to explain. It may be further contemplated, the namedreference labels may remain invariant throughout such dynamicalprocesses, even though the precise location, connection and relationshipbetween the part and the whole of the relevant named component maychange. Such invariance under dynamical processes makes named referencelabels an ideal component for use within explanations.

In another exemplary embodiment, the same invariance of named referencelabels under dynamical processes may be utilised by a BM to generatestable, long-term explanations of the dynamics occurring within an AImodel without having to recode knowledge or retrain explanatory methodswith each run of the dynamical process.

Any BMs associated with AI models that have had structural changes mayneed to update their dynamic explanations of the model's new behaviour,and undergo several consistency checks related to both the addition anddeletion, and in general, modifications to the underlying AI model.

The novel use of named reference labels in an exemplary embodimentwithin behavioural models and their association with evolving AI modelsthat have been modified using a suitable dynamical process, such asAutoXAI, model discovery, or model optimization process, may enable apractical implementation of neuro-symbolic constraints that may apply tosome up-stream meta-reasoning layer that has access to the statisticalor causal relationships between variables.

Named reference labels may also be used to integrate explainable AImodels and BMs with neuro-symbolic AI systems that can utilise symbolicrule expressions and be used to perform automated symbolic checks andverification that is impossible to do with a black-box AI system. Thecombination of a white-box AI system and a BM is particularly powerful,as it allows end-to-end understanding of the functioning of the AIsystem together with prediction and control of its behaviour.Furthermore, this combination allows for both static verification andlogical boundary-type checks of the AI system and the BM, together withdynamic checks, monitoring and verification of the AI system and the BM.

A number of events, triggers, constraints, and actions in a BM or BMHmay be deemed to be important or critical for the particular applicationdomain. Such importance may be determined either by the application orusage context, or via an external third party, such as a regulatory orlegal authority or an industry standard making body, that imposescertain mandatory constraints. These type of important or criticalconstraints may be referred to as anchor events, anchor triggers, anchorconstraints and anchor actions, respectively. Such anchor components areanalogous to the anchor terms, anchor variables, anchor nodes and anchoredges concepts within explainable models and within ExplanationStructure Models (ESMs).

A DEA may be utilized in a behavioral model framework in order toincrease the performance of the defined models. A DEA may containmultiple explainable architectures DEA_(m), such that m∈{x₁, x₂, . . . ,x_(n)}, where x∈{XAI, XNN, XTT, XRL, XSN, XMN, INN} or logicallyequivalent or similar architectures, and these models may be processedin parallel. The number of explainable architectures 1410 in thedistributed framework may be defined as DEA_(n). FIG. 13 illustrates ahigh-level architecture of an exemplary distributed training system,where DEA_(m) refers to the parallel DEA_(n) explainable models, whereDEA_(n) is the number of models in such framework.

A DEA may split the dataset into multiple subsets 1400 of data in orderto train the explainable architectures DEA_(x). The models trained in aDEA may be aggregated 1420 during the training phase by calculating theaverage (or weighted average) from the parallel models. The aggregatemodel may be formed based directly on the weights of the model, ratherthan from the result of the individual models. A DEA may be useful forlarge datasets where the training data cannot fit in the CPU/GPU orsimilar memory of a single machine.

A DEA may include hybrid models, such that the models in sucharchitecture are a mix of x, where x∈{XAI, XNN, XTT, XRL, XSN, XMN, INN}or logically equivalent or similar architectures. An exemplaryembodiment may implement multiple different models. In an exemplaryembodiment, one data part may implement an XNN while another data partof the same system may implement an XAI. The models may then be combinedto create an aggregate model. The aggregate model may be equal to x,where x∈{XAI, XNN, XTT, XRL, XSN, XMN, INN} or logically equivalent orsimilar architectures or may be a hybrid model implementing multipledifferent models. A distributed explainable architecture DEA mayincorporate multiple independent models where one model, once trained,can work independently without the need to rely on the full distributedarchitecture, which is optimized primarily for training purposes.

Conditional constraints may be set on the internal coefficients of anExplainable System 570, such as the explainable architecture x in a DEA.As illustrated in FIG. 14 , a conditional constraint c may be set on acoefficient that is located in a model m, where m∈{XAI, XNN, XTT, XRL,XSN, XMN, INN} or logically equivalent or similar architectures 1510.When such conditional constraint is set to true, an event e 1550 isfired in order to trigger 1560 an action a 1570. The action a mayperform a change in the status of a sub-component 1580 within the modelor raise an event e within the model. Actions BM_(a) may be triggered bya trigger t. A BM may contain a set number of triggers BM_(t)∈{t_(i), .. . , t_(n)}. A trigger t may be activated when a condition c set fortrigger, t_(c) is set to true. A trigger t may have multiple set ofconditions to be activated, such that t_(c)∈{c₁, . . . , c_(n)}.

Referring now to the exemplary embodiment in FIG. 15 , FIG. 15 mayillustrate an exemplary feed-forward distributed architecture forevents, triggers, and actions. In an exemplary embodiment, a distributedexplainable architecture DEA may have v explainable models, such thatDEA_(m), where m∈{x₁, x₂, . . . , x_(v)} andx∈{XAI,XNN,XTT,XRL,XSN,XMN,INN}. x₁ may have an action a 1670 that isbased on an internal trigger x_(1,t,1) that is activated when the fusionof two conditional constraints 1652 x_(1,c,1) and x_(1,c,2) are set totrue. x_(1,t) may refer to the triggers defined for the explainablemodel x₁, where t∈{t₁, . . . , t_(n)}. Trigger x_(1,t,2) may be based ona partial constraint 1660 of trigger x_(1,t,1), and the constraint beingx_(1,c,1). Hence, when the conditional constraints x_(1,c,1) andx_(1,c,2) are set to true, an event is fired to activate triggerx_(1,t,1), and an additional event is fired to activate x_(1,t,2). Thefusion of conditional constraints may be based on multiple models m,within a distributed explainable architecture DEA to trigger the desiredactions 1670 1580.

It is further contemplated that conditions, constraints, actions,triggers, and events may utilize a combination of abductive, inductive,deductive logic in conjunction with causal logic. Using inductive logic,BMs may predict future behavior based on generalized rules and knowledgethat may have been learnt fully automatically. Using deductive logic,BMs may predict behavior according to a combination of one or moreconditions or constraints. Using abductive logic, BMs may retrofit anobserved scenario to a known set of possible states in the BM or be ableto explain the currently observed behavior in a reasonably acceptablemanner. Abductive logic can also be useful in practical implementationsof diagnostic systems and can be used to aid in the diagnosis andtroubleshooting of AI systems using behavioral monitoring and predictivedata.

In an exemplary embodiment, a BM may use inductive logic to generalizesafety rules and constraints to unforeseen circumstances that may stillbe dangerous, nonetheless. It is further contemplated that usingdeductive logic, the BM may augment safety constraints with logicallyderived conclusions from the initial set of rules or constraints. It isfurther contemplated that using abductive logic, the BM may use theoutcomes of diagnostic and/or abductive results to refine the scope ofthe safety rules or constraints.

In an exemplary embodiment, BMs can also utilize multiple explainablemodels to fuse answers, model explanations and justifications comingfrom one or more models including but not limited to models hostedlocally, remotely, via a number of data transmission networks, on theedge, and embedded systems, which are further deployed as a single ordistributed model, or as ensembles.

In an exemplary embodiment, BMs can utilize an ensemble of explainablemodels or other suitable distributed system to improve performance inparallel or on suitable deployment architectures, such as cloudcomputing systems.

A behavioral model BM may contain multiple explainable architecturesBM_(x), such that BM_(x)∈{x₁, . . . , x_(n)}, and the output from aneuron node in an explainable architecture may be used to update aweight in another explainable architecture. The explainable architecturethat may include weights that are updated using this approach may be afast explainable architecture ƒ, where ƒ∈{F−XAI, F−XNN, F−INN, F−XTT,F−XRL}. A condition c in a behavioral model BM may be based on a fusionconstraint of weights and the output of a node of a fast explainablearchitecture, as shown in FIG. 11 . A condition c in a behavioral modelBM may be based on a fusion of constraints that may include a weightfrom a fast explainable architecture and a prediction output or acoefficient from an internal part from a normal explainablearchitecture. A condition c in a BM may be based solely on a fast weightin a fast explainable architecture.

In an exemplary embodiment, a BM may include an XNN explainablearchitecture x₁ 1200 and a fast XNN explainable architecture x₂ 1210.The x₁ 1200 prediction output layer of the prediction network may beconnected to a particular coefficient 1220 of the prediction network ofa fast XNN architecture x₂ 1210.

A Behavior Model Hierarchy (BMH) is a hierarchical framework that mayinclude two or more behavioral models. Behavioral models within a BMHmay be connected to each other or connected to one or more explainablesystems. BMH may be used to maintain scalability and create re-usablecomponents, as shown in FIG. 12 . BMH may include super-states BMH_(s)that communicate between its behavioral models BMH_(BM) 1320 1330 andgeneralized transitions 1340 to transition between such behavioralmodels BMH_(BM). Super-states BMH_(s) are a group of states that areutilized to prevent redundant transitions between normal states.Transitions between super-states BMH_(s) are referred to generalizedtransitions. BMH may utilize attributions and information, that isretrieved from internal states of such explainable architectures or fromthe output of the computation from such explainable architectures, toupdate the state of a super-state. Output may include any related metainformation of the explainable architectures or logically equivalent orsimilar architectures 1300. A Behavior Model Hierarchy BMH may alsocombine events e 1310, actions a, and triggers t from multiplebehavioral models BMH_(BM) in order to achieve a certain objective.Output 1350 may refer to the output, actions a, events e and triggers tactivated from the behavioral model BM_(k+1) 1330. A BMH may be definedas acyclic BMH or cyclical BMH. Cyclical BMH may refer to a BMH whichmay include feedback actions from a behavioral model BM to anotherbehavioral model BM within Behavior Model Hierarchy BMH. Acyclic BMH mayrefer to a BMH that does not contain feedback action loops betweenbehavioral models BMH_(BM).

In an exemplary embodiment, an autonomous system may be designed using aBMH. A BMH may include multiple behavioral models BMH_(BM), whereBMH_(BM)∈{BM₁, . . . , BM_(n)}. A BMH may include a machine learningmodel that is utilized by multiple behavioral models. An exemplaryautonomous system may have conditional constraints BMH_(c) on thewhite-box machine learning model. The activation of conditions BMH_(c)fire events BMH_(e) to triggers BMH_(t) that may be connected inmultiple behavioral models BM_(K) and BM_(k+1). The triggers BMH_(t) mayprovide feedback actions BMH_(af) to the machine learning model orterminal actions in behavioral model BMH_(at). An exemplary autonomoussystem using a BMH may include multiple machine learning models BMH_(x),where BMH_(x)∈{x₁, . . . , x_(n)}. In this exemplary case, conditions ofBehavior Model Hierarchy BMH may be fused together as a conditionalrequirement for a particular trigger in BMH_(t).

A BM or a BMH may require verification against a specification of thedesired behavior before being deployed in a system that requires asafety assurance. A model that achieved high accuracy might not becomeformally verified, since accuracy does not illustrate how much of theinternal logic was tested when validating the model on the unseendataset. Machine learning models may have blind spots for adversarialperturbations (Goodfellow et al., 2014). An adversarial perturbation mayrepresent input observations that are formed by applying small butspecific perturbations to observations that are utilized for predictionon the respective machine learning model. A BM may allow for triggers530, events 520, actions 550 540 and system components 560 to be basedon coefficients, output or other neuro-symbolic information contained inthe white-box explainable models (as shown in FIG. 4 ), in order tohandle unexpected prediction results, such as adversarial perturbations(Goodfellow et al., 2014) and take appropriate actions that preservesafety, while allowing the BM to be adaptable to unforeseencircumstances. In an exemplary embodiment, a BM may be used toinvestigate an unexpected prediction by analyzing the featureattributions of the input space for global and local bias.

Formal verification may verify the output of a model under specificconditions, in order to avoid costly mistakes. In an exemplaryembodiment, an autonomous vehicle system may be required to verify thatan output action for certain conditions must always be the same in orderto avoid fatal accidents. Hence, a BM or BMH may need to be formallyverified for particular specifications to ensure the decision of themodel when the model is deployed.

A BM that may include a deep learning model may be formally verifiedusing techniques such as Reluplex (Katz et al., 2017) to verify themodel for particular conditions on the input dataset and the outputvalue, in order to ensure that the conditions are satisfiable for themodel. Conditions c for black-box models may include constraints on theinput dimensions and the output dimensions of the model.

In another exemplary embodiment, a BM may be implemented usingneuromorphic hardware. The conditions, events, triggers, and actions ofa BM may also run entirely natively on appropriate neuromorphichardware. Explainable models, such as XNNs, that can also run nativelyon neuromorphic architectures can be seamlessly associated and linkedwith a BM entirely on neuromorphic hardware. Such an exemplaryembodiment may provide a practical solution for behavioral monitoring,assessment, identification, prediction, and control of neuromorphicdevices in a controlled, reliable, and replicable manner while ensuringsafety and adequate control and human oversight of neuromorphic systems.

In another exemplary embodiment, a BM may be implemented using a quantumprocessing system. It is contemplated that an BM implemented on aquantum processing system will have characteristics that are similar toclassical BM models with the addition of quantum specific extensions.For example, such an extension may allow for the specification ofquantum annealing effects and their correct interpretation. In anotherexample, an extension may allow for the correct interpretation ofmultiple qubit states, qubit basis states, mixed states, Ancilla bits,and other relevant quantum effects due to entanglement and/ordecoherence. In another example, an extension may allow for theintroduction of quantum logic specific operators and/or hardware logicgates within an XNN, such as quantum logic gates like CNOT(Controlled-NOT), CSWAP (Controlled-Swap or Fredkin gate), XX (IsingCoupling Gate XX), YY (Ising Coupling Gate YY), ZZ (Ising Coupling GateZZ) gates, Pauli gates, Hadamard gates, Toffoli gates and other relevantquantum logic operations that may be combined serially or in parallel.Such quantum-logic gates (or quantum gates) operate on a number ofqubits, acting as the quantum analogy of classical logic gates. The XX,YY, ZZ designation of the Ising gates are related to the Pauli X, Y, Zmatrices which are used to describe the interaction of a particle's spinwith an external electromagnetic field. Furthering these examples, suchquantum specific extensions may be implemented in various parts of theBM system, for example by having quantum extended versions ofconditions, events, triggers, and actions. It is further contemplatedthat such quantum extended versions of a BM may take advantage ofquantum effects, for example, to execute multiple actions, or evaluatemultiple conditions, or evaluate large systems of constraints insignificantly fewer processing steps needed than possible on a classicprocessing implementation.

Quantum Causal BMs can model quantum decoherence effects and theinherent uncertainties involved in quantum measurement. In such aQuantum BM, there may be multiple outcomes instead of a single outcomefor a Classical BM. Similarly, a Quantum EBM may have multiple expectedoutcomes instead of single outcomes in a Classical EBM. In a Quantum BM,the OBM represents the measurement, and hence collapse of quantumstates, and will thus represent similar information to a Classical OBM,although the actual results may be different due to the use of quantumlogic operations. A Quantum BM thus represents a practical way ofmonitoring, identifying, assessing, predicting, and controlling thebehaviour of a quantum AI model running on quantum computing device.Safety applications of Quantum BMs may take advantage of this byspeeding up constraint satisfaction searches and by considering multiplepredicted outcomes faster than what can be done with a Classical BM.Given the real-time nature of applications for BMs, the temporalreduction and performance increase given by Quantum BMs may be highlybeneficial.

In an exemplary embodiment, a Quantum BM may utilize suitable quantumalgorithms, such as those based on quantum Fourier transforms, amplitudeamplification, quantum walks and so on. In an exemplary Quantum BMembodiment, the Bernstein-Vazirani, Simon's algorithm or theDeutsch-Jozsa algorithm is utilized to predict and refine the boundaryconditions of the EBM. In another exemplary Quantum BM embodiment,Shor's algorithm, Quantum Phase estimation algorithm, Grover'salgorithm, Quantum Counting, Quantum Hamiltonian NAND trees, or the HHLalgorithm may be used to speed up the constraint, condition, event, andtrigger parts of Quantum BMs. In another exemplary Quantum BMembodiment, a hybrid solution may be utilized, such as the QAOAalgorithm, VQE eingensolver, CQE eingensolver, and quantum matrixinversion to speed up part of the processes involved, for example byusing Gaussian estimation processes, or linear system of equationssolvers that utilize quantum processing to give faster results.

In another exemplary embodiment, a BM may be used in conjunction with AImodels that process sequence data. Sequence data may include data pointswhich contain feature data in various sequential formats including, butnot limited to: 2D data, 3D data, multi-dimensional data arrays,transactional data, time series, digitised samples, sensor data, imagedata, hyper-spectral data, natural language text, video data, audiodata, haptic data, LIDAR data, RADAR data, SONAR data, and logicallyequivalent or similar sequential formats. Data points may have one ormore associated labels which may indicate the output value orclassification for a specific data point or a continuous ornon-continuous interval of data points. Data point sequences may resultfrom an internal and/or external process that may output a combinationof synthetic data points, perturbed data, sampled data, or transformeddata. Such data point sequences may be the input for BM constraintexpressions, conditions, events, triggers, and actions.

In an exemplary embodiment, a BM may be used to implement formalverification of an autonomous system to detect nearby pedestrians may bedescribed using constraints and/or rules such as “if a pedestrian isnear the autonomous system; and such pedestrian is coming from the left,perform a right turn”. An automated constraint and/or rule augmentationsystem may augment the verification statement by additional backgroundrules adopted from the world context, for example, by adding “where suchturn is determined by the system speed, predicted pedestrian speed andright-hand obstacle boundaries”. Such a verification problem can beconverted into a solvable system of equations, with parameters such asthe distance between the autonomous system and the pedestrian, D, givendist_(l)<D<dist_(u), where dist_(l) and dist_(u) are the lower and upperboundaries of such distance, and at angle θ, given ang_(l)<θ<ang_(u),where ang_(l) and ang_(u) are the lower and upper boundaries of theangle of the predicted pedestrian movement vector. Formal verificationtechniques may be utilized to verify that with such parameterconstraints, the prediction output for the right turn angle needs to bewithin boundaries o_(lower)<right_(turn)<o_(upper). Furthering theexample, such formal verification solutions may also be implemented inparallel with those based on observational, dynamic verification ofactual observed behavior, in this case, running on the autonomous systemitself. Continuing with the example, the autonomous system may alsooutput an explanation at a user-level like “Had to swerve strongly tothe right to avoid the pedestrian who ran in front of the car”.Adjectives in the explanation, such as “strongly” can be determined viasome appropriate classification method, value lookup table, fuzzy logicsystem, AI model or other appropriate text generation implementation.The autonomous system may also output more detailed explanations, forexample, at a diagnostic-level like “Car swerved strongly to the right(acceleration profile: angle=20°, speed delta=2.77 m/s) to avoidpedestrian (detected object: class=pedestrian, confidence=95%; objectvector (predicted): relative degree=90°, relative speed delta=2.22 m/s)collision (predicted collision=91%; predicted injury risk=HIGH;predicted vehicle component=front headlamp)”. The autonomous system mayalso utilize the diagnostic-level explanation, encoded in a suitableformat, such as a machine-readable method for encoding and transmittingXAI knowledge, to produce multimedia output, such as an animation on asuitable dashboard display or heads up display system or augmentedreality display. It is further contemplated that such information mayalso be transmitted over an appropriate Intelligent Transport System(ITS) network, such as a vehicle to vehicle (V2V) network, vehicular adhoc network (VANET), mobile ad hoc network (MANET) using an appropriateinter-vehicle communications (IVC) protocol such as an IEEE 802.11pcompatible protocol.

A BM may include an explainable architecture BM_(x), where x∈{XAI, XNN,XTT, XRL, XSN, XMN, INN} or logically equivalent or similararchitectures. An exemplary embodiment enables formal verificationconstraints to be set on the output or internal parts of suchexplainable architecture BM_(x). The output may include featureattributions for the input dimensions of the observation and partitioninformation for such observation. Internal parts of explainablearchitecture BM_(x) may include coefficients of the local model forobservation.

The execution sequence es of the behavioral model BM may refer to theexecution trace routed by the behavioral model transition functionbetween the defined Explainable System component 570 and othercomponents of the behavioral model BM. An execution sequence is uniqueif the behavioral model BM is a deterministic model. An executionsequence is not unique if the behavioral model BM is a non-deterministicmodel.

In the case of non-deterministic behavioral model, where the sequence isnot unique, when the same input is used for the behavioral modelmultiple execution traces are generated. Let input dimensions be definedas input_d, hence in a non-deterministic behavioral model,es_(input_d)={es₁, . . . ,es_(n)}. An exemplary embodiment enablesconditions BM, to be set on the execution sequences es_(input_d), suchthat events BM_(e) are fired to trigger an action a if conditions BM_(c)are true. In an exemplary embodiment, a system may rank the executionsequences es_(input_d) for the same input dimensions, according to theprobability likelihood of encountering the execution sequence in themodel.

In an exemplary embodiment of a non-deterministic behavioral model in anindustrial robotics application, for example, using a manufacturing toolheld by a robotic arm, a ranked sequence of possible robotic armlocations may be generated by the behavioral model and more intensewarnings may be issued if a human operator strays closer to a possiblecollision path with the robotic arm. If an imminent collision ispredicted to happen with a high probability, a temporary pause oremergency halt event may also be triggered to escalate the safetymechanism. It may be contemplated that such an embodiment may also beimplemented entirely in hardware for faster performance, for example, byutilizing an optimized XNN together with a Petri net-based BM that havebeen output as a dedicated hardware circuit. Further, it may becontemplated that such hardware embodiments may form part of the controlengineering circuitry of robotic machinery, appliances, and the likethat may require real-world interaction and control in a verifiable,certifiable, risk minimized, and safety assured manner.

An exemplary behavioral model may store the probability likelihoodinformation for its respective execution sequences es, known asprobabilistic execution sequences. A behavioral specification of abehavioral model BM may include a set of initial states and therespective execution traces. A behavioral specification may havecomplete coverage of all possible execution traces or at least partialcoverage if it does not include all possible execution traces.

In an exemplary embodiment, behavioral model BM₁ has the behavioralspecification SBM₁ and behavioral model BM₂ has the behavioralspecification SBM₂. The difference between the execution traces of BM₁and BM₂ is defined as ξ(SBM₁,SBM₂). The difference ξ(SBM₁,SBM₂) includeexecution sequences es, where es={es₁, . . . , es_(n)}, that are notidentical in SBM₁ and SBM₂.

The difference (SBM₁,SBM₂) between the behavioral specification SBM₁ ofa probabilistic behavioral model BM₁ and the behavioral specificationSBM₂ of non-probabilistic behavioral model BM₂ may be computed bydiscarding the probabilistic information of BM₁ and compare theexecution sequences es without using the probabilistic information.

The difference between the behavioral specification SBM₁ of aprobabilistic behavioral model BM₁ and the behavioral specification SBM₂of probabilistic behavioral model BM₂ is defined as ξ_(p)(SBM₁,SBM₂).The difference ξ_(p)(SBM₁,SBM₂) may include the probabilisticinformation of the execution sequences es, where es={es₁, . . . ,es_(n)}. The difference ξ_(p)(SBM₁,SBM₂) may be calculated using asuitable difference method such as subtraction between the probabilitylikelihoods of execution sequences es.

An exemplary embodiment may enable conditions of a behavioral modelhierarchy BMH_(c) to contain constraints on the difference between thebehavioral specification SBM₁ of behavioral model BM₁ and the behavioralspecification SBM₂ of behavioral model BM₂. In an exemplary embodiment,the behavioral model hierarchy conditions BMH_(c) may be based on athreshold th, where threshold th refers to the probability likelihooddifference between execution sequences es.

In an exemplary embodiment, an autonomous vehicle system may be based ona BMH. A BMH may include behavioral models BMH_(BM)∈{BM₁, . . .,BM_(i)}. An exemplary system may include conditional constraintsBMH_(c) such that the difference ξ_(p)(SEBM₁ SOBM₁) between thebehavioral specification of the expected behavioral model EBM₁ and thebehavioral specification of the observed behavioral model OBM₁ may beused to monitor for deviations between the expected behavioral modelEBM₁ and the empirical observations in execution sequences es of theobserved behavioral model OBM₁.

In an exemplary embodiment, BMs may be used to do constant monitoring ofAI models to detect anomalous behavior, detect instances of data driftand OOD instances, analyze and assess the behavior of AI models underOOD and anomalous instances, variation, deviation, performance andresource usage monitoring, phase-space, and other related monitoringactivities. BMs may also perform continuous certification of theassociated AI model, with an optional confidence/error interval,according to various criteria and raise an action when the certificationis in danger of being revoked.

An exemplary embodiment may assure the safety in the data managementstage of an exemplary machine learning lifecycle (Ashmore et al., 2019)through the underlying explainability architecture. The autonomoussystem may verify that the data is relevant to the task by analyzing thepatterns of the feature attributions of the input of the trainingdataset, the partitioning information of the explainable architecture,and the prediction data. A threshold could be utilized to monitor thedifference of the feature attributions between different inputs. A flagmay be raised in the system if the difference exceeds the definedthreshold. A threshold could be utilized to monitor the output of afunction ƒ, where ƒ may be the count of the number of observations ineach partition of the explainable architecture. If the threshold isexceeded on an unseen dataset, a flag may be raised in the system. Itmay be contemplated that a confidence interval or some other form ofprobability scoring metric is associated with the threshold, addingdiscrete or continuous threshold levels to an implemented system.

In an exemplary embodiment, a BM may be constructed to review the healthof the patients of a particular hospital. An explainable architecturemay be trained to measure the health of the patients using a CompleteBlood Count (CBC) blood test dataset. The BM may have conditionalconstraints BM, that are set on the explainable white-box architectureBM_(x), where x∈{XAI, XNN, XTT, XRL, XSN, XMN, INN} or logicallyequivalent or similar architectures. The conditional constraints BM, maybe set to monitor the feature attributions of the input features onunseen dataset and compare it with the feature attributions of thetraining dataset or other defined dataset. A function ƒ may be utilizedto measure the difference between the feature attributions on the unseendataset and on the training or defined dataset. The explainablearchitecture BM_(x) is trained on patients ongoing medical treatment A.A medical professional, using the explainable architecture BM_(x),predicts an unseen CBC blood tests dataset of patients ongoing adifferent medical treatment B. If the behavioral model conditionalconstraints BM, are activated, a warning is raised in the system usingevents BM_(e), triggers BM_(t) and actions BM_(a). In an exemplaryembodiment, a warning may be presented to the medical professionalstating that the dataset inserted for prediction might not representpatients that are ongoing the same medical treatment as the datasetutilized for training BM_(x).

An exemplary embodiment may assure the safety in the model learningstage and model verification stage, since an exemplary explainablearchitecture may also be interpretable, re-usable and validated on theselected machine learning algorithm. An exemplary embodiment providesmultiple levels of explanations: basic interpretation, explanatoryinterpretation, meta-explanatory interpretation.

An exemplary embodiment may be reusable and can be combined withadditional models to achieve the desired objective while retaining themodel explainability. The coefficients of the underlying explainablearchitecture may be visible to the system, allowing for human knowledgeinjection and system knowledge injection, and thus may offer an increasein the safety and assurance of an autonomous or semi-autonomous systemthat is using the behavioural model framework BM. Further interpretationand insights may be gained via human inspection of the results, theexplanation, justification, and the meta-explanation, leading to moreeffective collaboration between human users and the AI system.

Model reusability may be seen where the underlying explainablearchitecture is integrated within autoencoders and Convolutional NeuralNetwork (CNN) architectures and CNN-XNN and Backmap, where theunderlying explainable architecture is integrated within a CNNarchitecture. Model reusability may be found in the latent layer of anexplainable autoencoder (XAE) which may be used to condition the samplesof the generator on multiple criteria according to the input layer ofthe XAE, in an explainable generative adversarial network (XGAN). Anexemplary embodiment may be validated using performance metrics such asprecision, recall, AUC, AUC (PR) or accuracy, depending on the task ofthe machine learning model. A generalization error threshold may be set.In an exemplary embodiment, if the error threshold violates the expectedperformance criteria established by the behavioral model the process mayraise a warning or return to the data management stage. It may becontemplated that a confidence interval or some other form ofprobability scoring metric is associated with the threshold, addingdiscrete or continuous threshold levels to an implemented system. Formalverification may be utilized to check if the trained model complies witha set of formal properties, such as a mathematical proof.

In an exemplary embodiment, a behavioral model is constructed togenerate realistic blood test data of patients with a particular medicalcondition. An XGAN may be trained to create realistic blood test data.An XAE is trained, using the conditional feature constraints as input,and the latent layer is used as an input vector to the XGAN architectureto condition the type of blood test samples generated by the generator.The feature attributions generated for the explainable generator XG maybe backtracked, using an appropriate method such as the backmap orsuitable alternative, to the input of the XAE. The feature attributionsgenerated by the XAE may be compared with the backtracked featureattributions of the XG architecture using a function ƒ in order toidentify patterns between the conditional constraint latent compressionand the resultant conditioned samples. Such identified patterns arelinked to the behavioural model, allowing for formal verificationtechniques to be utilized to prove the correctness of certain formalspecification with respect to the input conditional constraints of theautoencoder. It is further contemplated that such a behavioural modelsolution may allow formal verification techniques, such as Reluplex(Katz et al., 2017) and other suitable techniques, to be utilized toverify the correctness of the given formal specification.

Ashmore et al. (2019) argued that an autonomous system must beprogrammed to handle unexpected prediction results from a machinelearning model. An exemplary embodiment may consist of assuring thesafety of a deployed white-box model within an autonomous orsemi-autonomous system. A behavioural model may allow for triggers 530,events 520, actions 550 540 and system components 560 to be based on thecoefficients of the white-box machine learning models, in order tohandle unexpected prediction results from the respective white-boxmachine learning models, and in a critical case, use a terminate action.As shown in FIG. 4 , this allows the behavioural model to be adaptableaccording to the output or internal coefficients of the white-boxmachine learning models. An exemplary embodiment may investigate anunexpected prediction by analyzing the feature attributions of the inputspace for global and local bias.

A method for classifying black-box model output, for triaging failureevents of autonomous systems is presented in (Zhou, 2018). Zhou does notprovide any interpretation of the classified output, as it does notutilize white-box explainable models. In our work, a behavioural modelBM that is used for triaging failure events of autonomous systems mayprovide three types of model interpretation for the classified output oftriaging failure events of autonomous systems. The explainable modelused in the BM, may provide three types of model interpretation: basicinterpretation, explanatory interpretation, and meta-explanatoryinterpretation. A basic interpretation may refer to a prediction outputo that can be understood by the sub-component. An explanatoryinterpretation may be represented by a 2-tuple vector <o, w> and mayrefer to a prediction output o combined with a model explanation w forthe predicted value, that can be understood by the sub-component. Amodel explanation may include coefficients θ of the explainablearchitecture x that may be utilized to explain the feature importance ofthe input features for a given observation. A meta-explanatoryinterpretation may be represented by a 3-tuple vector <o, w, j> and maycontain the prediction output o, the model explanation w andjustification of the model explanation j. The model justification j mayprovide additional information about the assumptions, processes anddecisions taken by the explanation system that were taken intoconsideration to produce the model explanation.

In an exemplary embodiment, a first vehicle failure event experienced bya first autonomous vehicle at a location has been classified by awhite-box model, a second autonomous vehicle can be directed to thelocation. The second autonomous vehicle may attempt to recreate asimilar driving scenario as to that which resulted in the first vehiclefailure event experienced by the first autonomous vehicle. Vehicle datacollected by the second autonomous vehicle at the location may beclassified using the white-box model and if a different failure typeclassification is provided for the vehicle data from the secondautonomous vehicle relative to the first autonomous vehicle, then ahuman understandable interpretation of the cause may be generated, fromthe explanation of the white-box model, using an interpretation filter.This contrasts with the work in (Zhou, 2018), where if the classifiedoutput of the second autonomous vehicle differs from the firstautonomous vehicle, it is assumed, without any explanation orjustification, that the issue that led to the first vehicle failureevent is isolated and/or specific to the first autonomous vehicle.Additionally, our BM may help generate a complete causal explanation,which is missing from the work in (Zhou, 2018).

Haynes et al. (2019) illustrate a method that uses black-box machinelearning models, such as static object classifiers, to predict thefuture locations of objects that are perceived by autonomous vehicles,without providing an explanation or interpretation of the predictedoutput. In this work, a behavioural model BM that utilizes explainablearchitectures to predict the future locations of objects that areperceived by autonomous vehicles may provide three types of modelinterpretation for the predicted output: a basic interpretation,explanatory interpretation, and meta-explanatory interpretation aspreviously disclosed above. Such explanations generated by BMs may makeall the difference between a trustworthy AI system (one that can explainthe underlying processes and steps taken regarding the predictions ofobjects) and an opaque non-trustworthy AI system (one that cannotexplain, such as in Haynes et al.) in a practical implementation.

Ashmore et al. (2019) argued that a generated ML model needs to exhibitan interpretable key property in domains where assurance is required.Ashmore et al. (2019) further argued that the generated ML model mayneed to be interpretable in order to be considered safe, whereinterpretability in this case may be seen via three main viewpoints:(i.) model interpretability, which is a measure of how interpretable isa model, together with its sub-components, structure and behavior; (ii.)output interpretability, which is a measure of how interpretable themodel's output is; and (iii.) logic interpretability, which is a measureof the optimality and correctness of the input-output relationship in amodel (based on the model's decision or choice). The model should becapable of outputting decisions it took to arrive at an output.Interpretable models allow for justification of the results generated bythe machine learning model and provide evidence for the output.Behavioural models may provide three types of model interpretation forthe predicted output: basic interpretation, explanatory interpretation,and meta-explanatory interpretation, as previously disclosed above.These three types of interpretation meet the full criteria set forth in(Ashmore et al., 2019), making this work suitable for practicalimplementations in application domains where assurance is required.

A Behavioral Model (BM) or Behavior Model Hierarchy (BMH) may beimplemented in Industry 4.0 automation systems, enabling smart factoriesto implement an explainable behavioral workflow based on white-boxmodels to ensure the safety of its operations in case of an abnormalevent or unexpected results from a white-box machine learning model. Inan exemplary embodiment, a Behavioral Model is used for a SupervisoryControl and Data Acquisition (SCADA) system to generate explainabledecisions and communicate any system issues to the operator, using thegenerated explanations to help mitigate downtime of the system. Theoperator may then pause the operation and view the generatedexplanations using a human-machine interface (HMI) to understand thecause of the issue, enabling the operator to act immediately on theexplanations, that may include information on the conditionalconstraints placed in the internal structure of an explainable system570, to prevent further loss of the product. In an alternative exemplaryembodiment, where the SCADA system is using black-box models in itsoperations, the operator may require additional time to understand why acertain erroneous output is being produced, and it might not always bepossible to arrive to an explainable conclusion, when using black-boxmodels in a system.

An observed behavioral model (OBM) consists of the base BM together withthe path traces that are observed during actual operation of the BM,including any out of distribution (OOD) data that takes the BM into anoperational space that was not part of the original training space. AnOBM may contain a mix of frequently executed paths and paths that areexecuted only occasionally. A BM may contain the predicted boundaries ofthe model associated with the BM. An OBM may contain actual boundariesof the model associated with the BM. It is contemplated that OBM actualboundary information may be utilized to update the BM predictedboundaries. It is further contemplated that OBM actual boundaryinformation may be utilized to update EBM predicted boundaries. Thepredicted boundaries may refer to, but not limited to, a sub-componentoutput of an explainable architecture x, where x∈{XAI, XNN, XTT, XRL,XSN, XMN, INN} or similar logically equivalent architectures. Thepredicted boundaries may also refer to a suitable transformation in anappropriate input-output space, logically equivalent, topologicallyequivalent or phase-space equivalent space based on the output of an AImodel associated with the BM. An OBM may identify non-optimal areas dueto the uncertainty in the associated model behavior. The identificationand eventual assessment of these non-optimal areas may be optimized byfine-tuning the predicted model boundaries. A narrow gap between thepredicted model boundary and the actual model boundary may indicate goodunderstanding and good fit of the BM. A BM may also potentially coverfuture predicted operational spaces and behavior for the associatedmodel for transactions and data that have not been observed yet. Suchnot-yet-observed areas are also referred to as Out-Of-Distribution (OOD)areas and are the focus of problems related to Zero-Shot, One-Shot andFew-Shot Learning. BMs may aid in the development of such systems byaiding in the identification of potential behavior that has not yet beenencountered or observed during actual operation of the associated model.A BM may also contain underspecified areas due to lack of modelcoverage. Neuro-symbolic symbols may help assure safety inunderspecified areas, by setting neuro-symbolic conditional constraintson the boundary of the global model. Weakness in the dataset may resultin a limited amount of information available, from the model pathtraces, or other forms of associated model information, in theconstruction of the BM predicted model boundaries, leading to incompletecoverage. The BM may also contain areas that correspond to dangerous ordisallowed areas. Neuro-symbolic symbols may assure the safety indangerous or disallowed areas by creating neuro-symbolic constraints onthe predicted boundaries. The impact of each node in the behavioralmodel may be calculated as the cumulative combination of the multiplecriteria measures that are applicable to the respective node and itssuccessor nodes (i.e., all possible child node path until the leafnodes), of the behavioral model. This impact can be calculated for bothtree and graph structures, enabling BMs to be applicable to bothstandard and graph explainable models, including XNNs with n-ary treelike partition hierarchies and XNNs with graph-like partitionhierarchies. The combination of multiple criteria measures is based onthe objective of the behavioral model.

In an exemplary embodiment, a BM may have multiple criteria measuresbased on model performance, bias reduction, and risk management. Thecombination of multiple criteria measures may be normalized byexpressing the total paths from the root as 1 and the rest of the pathsas a fraction of the total score bounded between [0 . . . 1]. It iscontemplated that a node discovery process in a behavioral model may usegame theory to discover the optimal nodes for the selected combinationof criteria measures. It is further contemplated that alternativemethods such as Multiple Objective Optimization (MOO), Pareto FrontMethods, Particle Swarm Optimization (PSO), Genetic Algorithms (GA),Bayesian Optimization, Evolutionary Strategies, Gradient Descenttechniques and Monte Carlo Simulation (MCS) may be used to discoveroptimal nodes for a given desired combination of criteria measures.

It may be contemplated that a behavioral model BM or hierarchy BMH maybe implemented and verified by on a combination of systems based on oneor more of the Temporal Logic of Actions, Abstract Machine Notation,Computation Tree Logic, and other suitable implementation methods thatcan formally represent modal logics, intuitionistic logics, and/orrelational semantics, including but not limited to Kripke semanticsand/or Alexandrov topologies.

In a further exemplary embodiment, a BM may be incorporated within aworkflow system that reads from the BM and writes back to the BM,including both processing data and event data. It is furthercontemplated that such BM and workflow combination may be furtherintegrated within a Robotic Process Automation (RPA) system or a DataLake system.

In another exemplary embodiment, BMs may be incorporated within asuitable risk identification, assessment, and mitigation framework, suchas that proposed by the ISO27001 model. It is also contemplated that BMsmay be incorporated within an Identify-Assess-Recommend-Resolve (IAR)framework that utilizes different metrics to identify issues, thenrelated metrics to assess the severity of the identified issue, followedby ranked and/or scored recommendations and finally coupled with adecision to execute such recommendation as part of a resolution plan. Itis further contemplated that such a BM may further incorporate aGoal-Plan-Action (GPA) system with the IAR framework.

In another exemplary embodiment, a BM may be implemented within anExplanation and Interpretation Generation System (EIGS), allowing theEIGS to add behavioural modelling, prediction, monitoring, behaviouralguarantees, and safety assurance to explainable and interpretable AImodels and to Explanation Filter Interpretation (EFI) systems.

Exemplary embodiments of Behavioural Models may allow modern AI systemsto reach a higher Evaluation Assurance Level (EAL) in the ISO/IEC 15408standard and also within the context of the Common Criteria RecognitionArrangement (CCRA). AI systems based on black-box methods or similararchitectures that do not allow for predictable and guaranteed behaviorcannot achieve an EAL rating higher than EAL1 and possibly barely beacceptable for EAL2. BMs, especially in conjunction with explainablemodels, allow for straightforward certification at the EAL1 to EAL4levels. The white-box nature of Behavioral Models, also allowcertification at the higher and more difficult to achieve EAL5 to EAL7levels, which is the highest level of verification and testing that canbe achieved within practical quality assurance frameworks commonly inuse world-wide. BM and explainable model-based AI system implementationscan thus allow users to acquire a higher level of confidence that thesystem's principal security features are reliably implemented, bymeeting specific assurance requirements. Typically, the functionalfeatures for each certified product or system are established in aSecurity Target document tailored for the EAL evaluation. Thus, asystem's fitness for purpose for a particular security applicationdepends on how well the features listed in the appropriate SecurityTarget fulfill the application's actual security requirements. BMs andtheir associated explainable models can be analyzed and predicted usingboth semi-formal and formal methods, which is something that is out ofscope for black-box systems. This inherent EAL related benefit for BMsallows hardware AI devices or related equivalent system to achievesuccessful EAL evaluations that are otherwise impossible to achieve. TheEAL related benefits also apply to other similar contexts, such as theUS FIPS 140-2, UK CESG Assisted Products Scheme (CAPS), the ISO/IEC27001 standard and other applicable national and international standardsand treaties.

An exemplary behavioral model framework may narrow the gap between theworld as imagined 420 and the world as observed 450. The underlyingexplainability architecture of a behavioral model framework may explainthe input dimensions of the explainable models using feature attributionand partitioning information and may provide multiple model explanationsas output.

Ashmore et al. (2019) argued that the deployment model in an autonomoussystem should be adaptable in order to allow updates and changes to bemade to the deployed model. Explanations can be attached to therespective actions, by utilizing conditional constraints, events, andtriggers, to assure the safety of an autonomous or semi-autonomoussystem.

An exemplary embodiment may utilize BMs that take input from acombination of human knowledge injection (HKI) and system-knowledgeinjection to update the weights and coefficients of the underlyingexplainable architecture, to further increase the safety of theautonomous or semi-autonomous system. It is further contemplated that BMtriggered actions may involve collaboration or a decision pointinvolving one or more human users.

An exemplary embodiment may be fully compatible with all current deeplearning libraries and architectures, allowing an embodiment to takeadvantage of all performance advancements available for deep learningsystems. An embodiment may further allow for interpretable models to betrained prior to the deployment on an autonomous system and to betrained while the system is already deployed. An exemplary framework mayallow for fusion of explainable models through partitioning and forunderstanding of the input features, through feature attributions andpartitioning information, to the explainable models by utilizingunderlying explainable architectures such as XNN or INN.

An alternative typical application of behavioral models is to integrateit with a combination of an Explainable Machine Learning System,Interpretable Machine Learning System, Explainer, Filter, Interpreter,Explanation Scaffolding, and Interpretation Scaffolding within thecontext of an Explanation and Interpretation Generation System (EIGS)and/or the Explanation-Filter-Interpretation (EFI) model.

In an exemplary embodiment, a BM may be used as the basis or part of apractical data privacy preserving AI system implementation. Data privacymay be violated intentionally or unintentionally by AI systems in anumber of scenarios: (i.) personal data from training datasetsunintentionally incorporated in AI models; (ii.) personal data can bere-extracted or re-created by analysing the model answers repeatedly;(iii.) personal data of certain uniquely identifiable groups may end upat a higher risk of identification; (iv.) model inversion and membershipinference techniques, that can associate model data via a unique key orsignature; (v.) other sources of information, such as public datasources, which may be combined with private information, may re-createor otherwise identify private information. The main data privacypreserving solutions for AI can be classified under four categories:(i.) differential privacy; (ii.) secure multi-party computation; (iii.)federated learning; (iv.) homomorphic encryption. Exemplary embodimentsof BM systems may enable practical implementations under all fourcategories.

In an exemplary privacy preserving solution (i.), differential privacy,the introduction of noise in the training data or some other suitablemeans of obfuscation, may be used to generate a controllable amount ofprivacy through a noise factor or ratio, in the BM. The noise level maybe a variable which the user may be able to supply or edit, where thenoise level may be implemented as a constraint and/or objective. Inprivacy preserving solution (ii.), secure multi-party computation (SMPC)may be used to obtain a correct answer while concealing partialinformation about data and may simultaneously compute the answer usingdata from one or more sources. Exemplary embodiments of BMs andexplainable models may extend SMPC protocols to apply to explanationgeneration apart from answer output. It is further contemplated thatexemplary embodiments of BMs can be analyzed and tested formally forsecurity and trust building purposes without revealing any privateinformation. A secure enclave may also be used to decrypt the data in aprotected space within the hardware processor, limiting the possibilitythat other parts of the system can access such data in clear text. Anend-to-end hardware implementation of a BM system with a secure enclavemay be rather resilient to most forms of data attacks. In privacypreserving solution (iii.), federated learning, a BM may be distributedacross various decentralized devices that hold only local data samples.The local data samples are not shared with other devices, thus limiting,but not completely eliminating, the privacy risk involved, and may beparticularly suitable for IoT or edge computing applications wheremessaging options are limited or constrained by the network topology,such as in a mesh network. In privacy preserving solution (iv.),homomorphic encryption, or homomorphic computing may be used to allowcomputation on encrypted data without either decrypting the data and,optionally, using encrypted explainable models. In an exemplaryembodiment of a BM using homomorphically encrypted data and ahomomorphically encrypted XNN, utilizing the CKKS protocol, a secret keyand a public key are generated. The public key is used for encryptionand can be shared, while the private key is used for decryption and mustbe kept secret, for example, in a secure hardware enclave or similarimplementation solution.

An exemplary explainable behavioral framework may provide a practicalsafety solution for virtual reality (VR), augmented reality (AR) andmetaverse applications, providing realistic safety boundaries of aphysical nature where applicable, or by utilizing a psychologicallyderived human-behavior model to identify and assess potentially harmfulsituations for the environment's VR/AR/metaverse participants.

An exemplary explainable behavioral framework may provide a practicalsafety solution for machine learning models and tasks to be deployed inan autonomous system or semi-autonomous system to behave within certainconditional boundaries to avoid unexpected results. An exemplarybehavioral framework may guarantee safety for the systems. Thus, safetyassurance and safe control and operation of an autonomous orsemi-autonomous system may be achieved without compromising theoperational efficiency of the system.

The foregoing description and accompanying figures illustrate theprinciples, preferred embodiments, and modes of operation of theinvention. However, the invention should not be construed as beinglimited to the particular embodiments discussed above. Additionalvariations of the embodiments discussed above will be appreciated bythose skilled in the art (for example, features associated with certainconfigurations of the invention may instead be associated with any otherconfigurations of the invention, as desired).

Therefore, the above-described embodiments should be regarded asillustrative rather than restrictive. Accordingly, it should beappreciated that variations to those embodiments can be made by thoseskilled in the art without departing from the scope of the invention asdefined by the following claims.

What is claimed is:
 1. A computer-implemented behavioral modeling methodfor handling, modeling, predicting, and verifying a behavior of a systemcomprising at least one explainable model, comprising executing on aprocessor the steps of: integrating at least one condition into at leastone of a plurality of execution sequences in a behavioral modelingarchitecture, each condition configured to trigger at least one actionif activated, and determining, for each of the plurality of executionsequences, a probability likelihood of encountering each of theplurality of execution sequences; identifying at least one explanationcorresponding to the at least one action in an action space, saidexplanation associated with at least one state and at least one reward,wherein at least one of the action space and the state space is discreteor continuous; predicting and storing at least one predefined predictedboundary associated with the at least one explainable model;determining, based on monitoring the at least one condition, that the atleast one condition has been met and that the at least one action pairedwith the at least one condition has been triggered; determining anactual boundary demonstrated by triggering of the at least onecondition, and, based on observation of the actual boundary, identifyinga gap between the predicted predefined boundary of the at least oneexplainable model and the actual boundary; updating the at least onepredefined predicted boundary of the at least one explainable modelbased on information observed during monitoring of the at least onecondition and based on the gap identified between the predictedpredefined boundary of the at least one explainable model and the actualboundary.
 2. The method of claim 1, wherein the system comprises anobserved behavioral model (OBM) comprising the at least one explainablemodel and further comprising a plurality of actual boundaries associatedwith the at least one explainable model, the plurality of actualboundaries including the actual boundary demonstrated by triggering ofthe at least one condition.
 3. The method of claim 1, wherein the atleast one explainable model comprises a non-deterministic behavioralmodel, said non-deterministic behavioral model comprising a ranked orderof the plurality of activation sequences based on the probabilitylikelihood of encountering each of the plurality of execution sequences.4. The method of claim 1, wherein identifying the at least oneexplanation corresponding to the at least one action in the action spacecomprises identifying an associate explanation integrating an action andan explanation of the action.
 5. The method of claim 1, wherein updatingthe at least one predefined predicted boundary comprises at least one ofupdating at least one coefficient defined within the at least oneexplainable model, or creating a rule or partition within an internalstate of the at least one explainable model.
 6. The method of claim 1,wherein the at least one explainable model comprises at least one of aprediction network and a conditional network; and wherein one or moreconstraints are provided on the at least one explainable modelencompassing the at least one of the prediction network and theconditional network.
 7. The method of claim 1, wherein the at least oneexplainable model includes a feature attribution network; and whereinone or more constraints are provided on the explainable modelencompassing the feature attribution network, said one or moreconstraints configured to monitor one or more feature attributionsassociated with an input dataset and compare the one or more featureattributions associated with the input dataset with one or more featureattributions associated with a predetermined dataset.
 8. The method ofclaim 7, wherein the one or more constraints configured to monitor theone or more feature attributions associated with the input dataset areconfigured to provide an alert in a case where a difference between theone or more feature attributions associated with the input dataset andthe one or more feature attributions associated with a predetermineddataset is above a predetermined level.
 9. The method of claim 1,further comprising labeling one or more components within the at leastone explainable model with one or more reference labels, wherein eachreference label indicates an activation path leading to the labeledcomponent.
 10. The method of claim 9, wherein each reference labelcomprises one or more of: symbolic expressions provided in one ofconjunctive normal form, disjunctive normal form, or first order logic;and wherein the one or more reference labels include at least one linkto an external taxonomy, ontology, or model.
 11. The method of claim 9,further comprising outputting one or more explanations based on one ormore reference labels, wherein the one or more reference labels comprisean interpretable path trace for one or more components associated withthe reference labels.
 12. The method of claim 1, wherein the system is aphysical hardware device comprising one or more moving elements, andwherein observation of the actual boundary comprises observation of amovement action executed by the one or more moving elements.
 13. Themethod of claim 12, wherein the system further comprises one or moresensors linked to the physical hardware device, and wherein observationof the actual boundary comprises receiving and interpreting data fromthe one or more sensors.
 14. The method of claim 1, further comprisingconverting an input of the system to a set of hidden features,identifying feature weights from the hidden features, combining thefeature weights with the transformed input, and extracting anattribution of each hidden feature based on the combined feature weightsand hidden features.
 15. The method of claim 1, further comprisingcreating a plurality of partitions within the system, wherein eachpartition is associated with a label, a graph, a hypergraph, or asimplicial complex; and associating a plurality of neuro-symbolicconstraints with the partitions, wherein the neural-symbolic constraintsassociated with the partitions comprise one or more of: symbolic rulesor a system of symbolic expressions, polynomial expressions, conditionaland non-conditional probability distributions, joint probabilitydistributions, state-space and phase-space transforms,integer/real/complex/quaternion/octonion transforms, Fourier transforms,Walsh functions, Haar and non-Haar wavelets, generalized L2 functions,fractal-based transforms, Hadamard transforms, Type 1 and Type 2 fuzzylogic, topological transforms of Kolmogorov/Frechet/Hausdorff/Tychonoffspaces, and difference analysis.
 16. The method of claim 15, whereineach partition comprises a local model with one or more rules, and thepartitions are aggregated to form a global model.
 17. The method ofclaim 1, wherein the behavioral modeling architecture is implemented onone or more Finite State Machines, Petri Nets, Robotic ProcessAutomation (RPA) systems, Actor Models, workflow systems, or quantumprocessing systems.
 18. The method of claim 1, further comprising:providing a causal model separate from the at least one explainablemodel, said at least one causal model configured to implement causallogic and at least one of deductive logic, abductive logic, andinductive logic; and triggering at least one of an event or an actionbased on an output of the causal model.
 19. A system comprising aprocessor and a memory and configured to implement at least oneexplainable model, wherein the system is configured to execute, on theprocessor, steps of: integrating at least one condition into at leastone of a plurality of execution sequences in a behavioral modelingarchitecture, each condition configured to trigger at least one actionif activated, and determining, for each of the plurality of executionsequences, a probability likelihood of encountering each of theplurality of execution sequences; identifying at least one explanationcorresponding to the at least one action in an action space, saidexplanation associated with at least one state and at least one reward,wherein at least one of the action space and the state space is discreteor continuous; predicting and storing at least one predefined predictedboundary associated with the at least one explainable model; monitoringthe at least one condition, and determining that the at least onecondition has been met and that the at least one action paired with theat least one condition has been triggered; observing an actual boundarydemonstrated by triggering of the at least one condition, and, based onobservation of the actual boundary, identifying a gap between thepredicted predefined boundary of the at least one explainable model andthe actual boundary; updating the at least one predefined predictedboundary of the at least one explainable model based on informationobserved during monitoring of the at least one condition and based onthe gap identified between the predicted predefined boundary of the atleast one explainable model and the actual boundary.
 20. Anon-transitory computer-readable medium comprising program code that,when executed by a processor of a system configured to provide at leastone explainable model, is configured to cause the processor to executesteps of: integrating at least one condition into at least one of aplurality of execution sequences in a behavioral modeling architecture,each condition configured to trigger at least one action if activated,and determining, for each of the plurality of execution sequences, aprobability likelihood of encountering each of the plurality ofexecution sequences; identifying at least one explanation correspondingto the at least one action in an action space, said explanationassociated with at least one state and at least one reward, wherein atleast one of the action space and the state space is discrete orcontinuous; predicting and storing at least one predefined predictedboundary associated with the at least one explainable model; monitoringthe at least one condition, and determining that the at least onecondition has been met and that the at least one action paired with theat least one condition has been triggered; observing an actual boundarydemonstrated by triggering of the at least one condition, and, based onobservation of the actual boundary, identifying a gap between thepredicted predefined boundary of the at least one explainable model andthe actual boundary; updating the at least one predefined predictedboundary of the at least one explainable model based on informationobserved during monitoring of the at least one condition and based onthe gap identified between the predicted predefined boundary of the atleast one explainable model and the actual boundary.